Knowledge and Information Systems

, Volume 58, Issue 1, pp 35–58 | Cite as

Rating the skill of synthetic agents in competitive multi-agent environments

  • Chairi KiourtEmail author
  • Dimitris Kalles
  • George Pavlidis
Regular Paper


A very effective and promising approach to simulate real-life conditions in multi-agent virtual environments with intelligent agents is to introduce social parameters and dynamics. Introduction of social parameters in such settings reshapes the overall performance of the synthetic agents, so a new challenge of reconsidering the methods to assess agents’ evolution emerges. In a number of studies regarding such environments, the rating of the agents is being considered in terms of metrics (or measures or simple grading) designed for humans, such as Elo and Glicko. In this paper, we present a large-scale evaluation of existing rating methods and a proposal for a new rating approach named Relative Skill-Level Estimator (RSLE), which can be regarded as a base for developing rating systems for multi-agent systems. The presented comparative study highlights an inconsistency in rating synthetic agents in the context described by the widely used methods and demonstrates the efficiency of the new RSLE.


Competitive social environments Multi-agent systems Synthetic agents Rating systems 


  1. 1.
    Al-Khateeb B, Kendall G (2011) Introducing a Round Robin tournament into evolutionary individual and social learning checkers. In: Developments in E-systems engineering, Dubai, United Arab Emirates, pp 294–299.
  2. 2.
    Attle S, Baker B (2007) Cooperative learning in a competitive environment: classroom applications. Int J Teach Learn High Educ 19(1):77–83Google Scholar
  3. 3.
    Basili V, Caldiera G, Rombach HD (1994) Goal question metric approach. In: Encyclopedia of software engineering, pp 528–532Google Scholar
  4. 4.
    Busoniu L, Babuska R, De Schutter B (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybern Part C Appl Rev 38(2):156–172. CrossRefGoogle Scholar
  5. 5.
    Caballero A, Botia J, Gomez-Skarmeta A (2011) Using cognitive agents in social simulations. Eng Appl Artif Intell 24(7):1098–1109. CrossRefGoogle Scholar
  6. 6.
    Calvo R (1998) Valencia Spain: the cradle of European chess. In: Presentation to the CCI, Vienna, AustriaGoogle Scholar
  7. 7.
    Di Bitonto P, Laterza M, Rosell T, Rossano V (2010) An evaluation method for multi-agent systems. In: Agent and multi-agent systems: technologies and applications, pp 32-41.
  8. 8.
    Edelkamp S, Kissmann P (2008) Symbolic classication of general two-player games. In: Technical report. Technische Universitat, DortmundGoogle Scholar
  9. 9.
    Elo AE (1978) The rating of chess players, past and present. Arco Publishing, New YorkGoogle Scholar
  10. 10.
    Ferber J (1999) Multi-agent systems: an introduction to distributed artificial intelligence. Addison-Wesley, BostonGoogle Scholar
  11. 11.
    Ferber J, Gutknecht O, Michel F (2004) From agents to organizations: an organizational view of multi-agent systems. In: In Agent-oriented software engineering IV, LNCS 2935. Springer, Berlin, pp 214–230.
  12. 12.
    Francisca T, Castro AM, Ana PR, Eugnio O (2013) Multi-agent learning in both cooperative and competitive environments. In: 16th Portuguese conference on artificial intelligence. Azores, Portugal, pp 400–411Google Scholar
  13. 13.
    Friedrich H, Rogalla O, Dillmann R (1998) Integrating skills into multi-agent systems. J Intell Manuf 9(2):119–127. CrossRefGoogle Scholar
  14. 14.
    Gilbert N, Troitzsch KG (2005) Simulation for the social scientist, 2nd edn. Open University Press, New YorkGoogle Scholar
  15. 15.
    Glickman EM (1995) A comprehensive guide to chess ratings. Am Chess J 3:59–102Google Scholar
  16. 16.
    Glickman EM (1999) Parameter estimation in large dynamic paired comparison experiments. Appl Stat 48:377–394zbMATHGoogle Scholar
  17. 17.
    Glickman EM, Albyn JC (1999) Rating the chess rating system. Chance 12(2):21–28Google Scholar
  18. 18.
    Harkness K (1967) Official chess handbook. McKay, PhiladelphiaGoogle Scholar
  19. 19.
    Herbrich R, Minka T, Graepel T (2007) TrueSkill(TM): a Bayesian skill rating system. Adv Neural Inf Process Syst 19:269–576Google Scholar
  20. 20.
    Hoen JP, Tuyls K, Panait L, Luke S, La Poutr JA (2005) An overview of cooperative and competitive multiagent learning. In: Proceedings of the first international conference on learning and adaption in multi-agent systems, Utrecht, The Netherlands, pp 1–46.
  21. 21.
    Kalles D (2007) Measuring expert impact on learning how to play a board game. In: IFIP conference on artificial intelligence applications and innovations, Athens, GreeceGoogle Scholar
  22. 22.
    Kalles D, Kanellopoulos P (2001) On verifying game design and playing strategies using reinforcement learning. In: Proceedings of ACM symposium on applied computing, special track on artificial intelligence and computation logic. ACM, Las Vegas, pp 6–11Google Scholar
  23. 23.
    Kapetanakis S, Kudenko D (2002) Reinforcement learning of coordination in cooperative multi-agent system. In: Proceeding of eighteenth national conference on artificial intelligence. Edmonton, Alberta, pp 326–331Google Scholar
  24. 24.
    Kendall M (1936) A new measure of rank correlation. Biometrica 30(1/2):81–93zbMATHCrossRefGoogle Scholar
  25. 25.
    Kiourt C, Kalles D (2012) Social reinforcement learning in game playing. In: IEEE international conference on tools with artificial intelligence, Athens, Greece, pp 322–326.
  26. 26.
    Kiourt C, Kalles D (2016) Synthetic learning agents in game-playing social environments. Adapt Behav 24(6):411–427. CrossRefGoogle Scholar
  27. 27.
    Kiourt C, Pavlidis G, Kalles D (2016a) Human rating methods on multi-agent systems. In: Multi-agent systems and agreement technologies: 13th European conference, EUMAS 2015, and third international conference, AT 2015, Athens, Greece. Springer, pp 129–136.
  28. 28.
    Kiourt C, Pavlidis G, Kalles D (2016b) ReSkill: relative skill-level calculation system. In: 9th hellenic conference on artificial intelligence (SETN2016), Thessaloniki, Greece, p 39.
  29. 29.
    Kiourt C, Kalles D and Kanellopoulos P (2017) How game complexity affects the playing behavior of synthetic agents. In: 15th European conference on multi-agent systems (EUMAS2017), FranceGoogle Scholar
  30. 30.
    Kotb Y, Beauchemin S, Barron J (2012) Workflow nets for multiagent cooperation. IEEE Trans Autom Sci Eng 9(1):198–203. CrossRefGoogle Scholar
  31. 31.
    Langville NA, Meyer DC (2012) Who’s #1?: The science of rating and ranking. Princeton University Press, PrincetonzbMATHCrossRefGoogle Scholar
  32. 32.
    Liang, Shi W (2005) Performance evaluation of rating aggregation algorithms in reputation systems. In: International conference on collaborative computing: networking, applications and worksharing, p 10.
  33. 33.
    Logan Y, Kagan T (2013) Elo ratings for structural credit assignment in multiagent systems. In: Twenty-seventh AAAI conference on artificial intelligence (late-breaking developments)Google Scholar
  34. 34.
    March JG (1991) Exploration and exploitation in organizational learning. Organ Sci 2(1):71–87MathSciNetCrossRefGoogle Scholar
  35. 35.
    Marivate VN, Marwala T (2008) Social learning methods in board game agents. In: IEEE symposium computational intelligence and games, Perth, Australia, pp 323–328.
  36. 36.
    Nikolenko S, Sirotkin A (2011) A new bayesian rating system for team competitions. In: Proceedings of the 28th international conference on machine learning, Bellevue, Washington, pp 601–608Google Scholar
  37. 37.
    Panait L, Luke S (2005) Cooperative multi-agent learning: the state of the art. Auton Agent Multi-Agent Syst 11(3):387–434. CrossRefGoogle Scholar
  38. 38.
    Pieter JH, Tuyls K, Panait L, Luke S (2006) An overview of cooperative and competitive multiagent learning. In: Learning and adaption in multi-agent systems. Lecture notes in computer science, vol 3898, pp 1–46.
  39. 39.
    Ruohomaa S, Kutvonen L, Koutrouli E (2007) Reputation management survey. In: ARES’07: the second international conference on availability, reliability and security, pp 103–111.
  40. 40.
    Russell S, Norving P (2010) Artificial intelligence: a modern approach. Prentice-Hall Inc, Upper Saddle RiverGoogle Scholar
  41. 41.
    Shoham Y, Leyton KB (2009) Multiagent systems: algorithmic, game-theoretic and logical foundations. Cambridge University Press, New YorkzbMATHGoogle Scholar
  42. 42.
    Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101CrossRefGoogle Scholar
  43. 43.
    Sutton R (1988) Learning to predict by the methods of temporal differences. Mach Learn 3(1):9–44. CrossRefGoogle Scholar
  44. 44.
    Sutton R, Barto A (1998) Reinforcement learning—an introduction. MIT Press, CambridgeCrossRefGoogle Scholar
  45. 45.
    Tampuu A, Matiisen T, Kodelja D, Kuzovkin I, Korjus K, Aru J, Vicente R (2017) Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE 12(4):e0172395. CrossRefGoogle Scholar
  46. 46.
    Tesauro G (1992) Practical issues in temporal difference learning. Mach Learn 4:257–277. zbMATHCrossRefGoogle Scholar
  47. 47.
    Tesauro G (1995) Temporal difference learning and TD-gammon. Commun ACM 38(3):56–68. CrossRefGoogle Scholar
  48. 48.
    Tesauro G (2002) Programming backgammon using self-teaching neural nets. Artif Intell 134:181–199. zbMATHCrossRefGoogle Scholar
  49. 49.
    Tromp J (2008) Solving connect-4 on medium board sizes. ICGA J 31(1):110–112CrossRefGoogle Scholar
  50. 50.
    Weng RC, Lin C (2011) A bayesian approximation method for online ranking. J Mach Learn Res 12:267–300MathSciNetzbMATHGoogle Scholar
  51. 51.
    Wilensky U (1999) NetLogo itself. In: Center for connected learning and computer-based modeling. Northwestern University, EvanstonGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Science and TechnologyHellenic Open UniversityPatrasGreece
  2. 2.“Athena” Research CentreUniversity Campus at KimmeriaXanthiGreece

Personalised recommendations