Multiagent Learning through Neuroevolution

  • Risto Miikkulainen
  • Eliana Feasley
  • Leif Johnson
  • Igor Karpov
  • Padmini Rajagopalan
  • Aditya Rawal
  • Wesley Tansey
Chapter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7311)

Abstract

Neuroevolution is a promising approach for constructing intelligent agents in many complex tasks such as games, robotics, and decision making. It is also well suited for evolving team behavior for many multiagent tasks. However, new challenges and opportunities emerge in such tasks, including facilitating cooperation through reward sharing and communication, accelerating evolution through social learning, and measuring how good the resulting solutions are. This paper reviews recent progress in these three areas, and suggests avenues for future work.

Keywords

Neuroevolution neural networks intelligent agents games 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Acerbi, A., Nolfi, S.: Social learning and cultural evolution in embodied and situated agents. In: IEEE Symposium on Artificial Life, ALIFE 2007, pp. 333–340. IEEE (2007)Google Scholar
  2. 2.
    Acerbi, A., Parisi, D.: Cultural transmission between and within generations. Journal of Artificial Societies and Social Simulation 9(1) (2006)Google Scholar
  3. 3.
    Arneson, R.: Egalitarianism. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Stanford University (2009)Google Scholar
  4. 4.
    Boehm, C.: Hierarchy in the forest: The evolution of egalitarian behavior. Harvard Univ. Pr. (2001)Google Scholar
  5. 5.
    Byrne, R., Whiten, A.: Machiavellian Intelligence: Social Expertise and the Evolution of Intellect in Monkeys, Apes, and Humans. Oxford University Press, USA (1989)Google Scholar
  6. 6.
    Czirok, A., Vicsek, T.: Collective behavior of interacting self-propelled particles. Physica A 281, 17–29 (2000)CrossRefGoogle Scholar
  7. 7.
    de Oca, M., Stutzle, T., Van den Enden, K., Dorigo, M.: Incremental social learning in particle swarms. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41(2), 368–384 (2011)CrossRefGoogle Scholar
  8. 8.
    Denaro, D., Parisi, D.: Cultural evolution in a population of neural networks. In: Marinaro, M., Tagliaferri, R. (eds.) Neural Nets Wirn 1996, pp. 100–111. Springer, Newyork (1996)Google Scholar
  9. 9.
    Dorigo, M., Maniezzo, V., Colorni, A.: Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics–Part B 26(1), 29–41 (1996)CrossRefGoogle Scholar
  10. 10.
    Erda, L., Whiten, A., Mellars, P., Gibson, K.: Egalitarianism and Machiavellian intelligence in human evolution. McDonald Institute for Archaeological Research (1996)Google Scholar
  11. 11.
    Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: From architectures to learning. Evolutionary Intelligence 1, 47–62 (2008)CrossRefGoogle Scholar
  12. 12.
    Floreano, D., Urzelai, J.: Evolutionary robots with on-line self-organization and behavioral fitness. Neural Networks 13, 431–4434 (2000)CrossRefGoogle Scholar
  13. 13.
    Fogel, D.B.: Blondie24: Playing at the Edge of AI. Morgan Kaufmann, San Francisco (2001)Google Scholar
  14. 14.
    Fogel, L., Owens, A., Walsh, M.: Artificial intelligence through simulated evolution. John Wiley (1966)Google Scholar
  15. 15.
    Gomez, F., Miikkulainen, R.: Incremental evolution of complex general behavior. Adaptive Behavior, 317–342 (1997)Google Scholar
  16. 16.
    Gomez, F., Miikkulainen, R.: Active guidance for a finless rocket using neuroevolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 2084–2095. Morgan Kaufmann, San Francisco (2003)Google Scholar
  17. 17.
    Haasdijk, E., Vogt, P., Eiben, A.: Social learning in population-based adaptive systems. In: IEEE Congress on Evolutionary Computation, CEC 2008 (IEEE World Congress on Computational Intelligence), pp. 1386–1392. IEEE (2008)Google Scholar
  18. 18.
    Herrmann, E., Call, J., Hernández-Lloreda, M., Hare, B., Tomasello, M.: Humans have evolved specialized skills of social cognition: The cultural intelligence hypothesis. Science 317(5843), 1360 (2007)CrossRefGoogle Scholar
  19. 19.
    Humphrey, N.: The social function of intellect. Growing Points in Ethology, 303–317 (1976)Google Scholar
  20. 20.
    Karpov, I.V., Sheblak, J., Miikkulainen, R.: OpenNERO: A game platform for AI research and education. In: Proceedings of the Fourth Artificial Intelligence and Interactive Digital Entertainment Conference (2008)Google Scholar
  21. 21.
    Kohl, N., Stanley, K.O., Miikkulainen, R., Samples, M., Sherony, R.: Evolving a real-world vehicle warning system. In: Proceedings of the Genetic and Evolutionary Computation Conference (2006)Google Scholar
  22. 22.
    Kubota, N., Shimojima, K., Fukuda, T.: Virus-evolutionary genetic algorithm -coevolution of planar grid model. In: Proceedings of the Fifth IEEE International Conference on Fuzzy Systems (FUZZ IEEE 1996), pp. 8–11 (1996)Google Scholar
  23. 23.
    Lipson, H., Pollack, J.B.: Automatic design and manufacture of robotic lifeforms. Nature 406, 974–978 (2000)CrossRefGoogle Scholar
  24. 24.
    Miikkulainen, R.: Neuroevolution. In: Encyclopedia of Machine Learning. Springer, Berlin (2010)Google Scholar
  25. 25.
    Mitchell, M., Thomure, M.D., Williams, N.L.: The role of space in the success of coevolutionary learning. In: Artificial Life X: Proceedings of the Tenth International Conference on the Simulation and Synthesis of Living Systems, pp. 118–124 (2006)Google Scholar
  26. 26.
    Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive coevolution. Evolutionary Computation 5(4), 373–399 (1997)CrossRefGoogle Scholar
  27. 27.
    Nowak, M.A.: Five rules for the evolution of cooperation. Science 314, 1560–1563 (2006)CrossRefGoogle Scholar
  28. 28.
    Parisi, D.: Cultural evolution in neural networks. IEEE Expert 12(4), 9–14 (1997)CrossRefGoogle Scholar
  29. 29.
    Pérez-Uribe, A., Floreano, D., Keller, L.: Effects of Group Composition and Level of Selection in the Evolution of Cooperation in Artificial Ants. In: Banzhaf, W., Ziegler, J., Christaller, T., Dittrich, P., Kim, J.T. (eds.) ECAL 2003. LNCS (LNAI), vol. 2801, pp. 128–137. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  30. 30.
    Rawal, A., Rajagopalan, P., Miikkulainen, R.: Constructing competitive and cooperative agent behavior using coevolution. In: IEEE Conference on Computational Intelligence and Games (CIG 2010) (August 2010)Google Scholar
  31. 31.
    Reynolds, C.W.: Flocks, herds, and schools: A distributed behavioral model. In: Computer Graphics (SIGGRAPH 1987 Conference Proceedings), vol. 21(4), pp. 25–34 (1987)Google Scholar
  32. 32.
    Reynolds, R.: An introduction to cultural algorithms. In: Proceedings of the Third Annual Conference on Evolutionary Programming, pp. 131–139. World Scientific (1994)Google Scholar
  33. 33.
    Roeva, O., Pencheva, T., Tzonkov, S., Arndt, M., Hitzmann, B., Kleist, S., Miksch, G., Friehs, K., Flaschel, E.: Multiple model approach to modelling of escherichia coli fed-batch cultivation extracellular production of bacterial phytase. Electronic Journal of Biotechnology 10(4), 592–603 (2007)CrossRefGoogle Scholar
  34. 34.
    Rumelhart, D., Hinton, G., Williams, R.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)CrossRefGoogle Scholar
  35. 35.
    Schrum, J., Miikkulainen, R.: Evolving agent behavior in multiobjective domains using fitness-based shaping. In: Proceedings of the Genetic and Evolutionary Computation Conference (2010)Google Scholar
  36. 36.
    Schrum, J., Miikkulainen, R.: Evolving multimodal networks for multitask games. In: Proceedings of the IEEE Conference on Computational Intelligence and Games (CIG 2011), pp. 102–109. IEEE, Seoul (2011)CrossRefGoogle Scholar
  37. 37.
    Seno, H.: A density-dependent diffusion model of shoaling of nesting fish. Ecol. Modell. 51, 217–226 (1990)CrossRefGoogle Scholar
  38. 38.
    Simpson, G.: The baldwin effect. Evolution 7(2), 110–117 (1953)CrossRefGoogle Scholar
  39. 39.
    Stanley, K., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary Computation 10(2), 99–127 (2002)CrossRefGoogle Scholar
  40. 40.
    Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the NERO video game. IEEE Transactions on Evolutionary Computation 9(6), 653–668 (2005)CrossRefGoogle Scholar
  41. 41.
    Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the NERO video game. IEEE Transactions on Evolutionary Computation 9(6), 653–668 (2005)CrossRefGoogle Scholar
  42. 42.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  43. 43.
    Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurrency - Practice and Experience 17(2-4), 323–356 (2005)CrossRefGoogle Scholar
  44. 44.
    Tomasello, M.: The cultural origins of human cognition. Harvard Univ. Pr. (1999)Google Scholar
  45. 45.
    Valsalam, V., Miikkulainen, R.: Evolving symmetry for modular system design. IEEE Transactions on Evolutionary Computation 15, 368–386 (2011)CrossRefGoogle Scholar
  46. 46.
    Vogt, P., Haasdijk, E.: Modeling social learning of language and skills. Artificial Life 16(4), 289–309 (2010)CrossRefGoogle Scholar
  47. 47.
    Waibel, M., Floreano, D., Magnenat, S., Keller, L.: Division of labour and colony efficiency in social insects: effects of interactions between genetic architecture, colony kin structure and rate of perturbations. In: Proceedings of the Royal Society B, vol. 273, pp. 1815–1823 (2006)Google Scholar
  48. 48.
    Watkins, C., Dayan, P.: Q-learning. Machine Learning 8(3), 279–292 (1992)MATHGoogle Scholar
  49. 49.
    Werbos, P.: Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10), 1550–1560 (1990)CrossRefGoogle Scholar
  50. 50.
    Whiteson, S., Stone, P.: Evolutionary function approximation for reinforcement learning. The Journal of Machine Learning Research 7, 877–917 (2006)MathSciNetMATHGoogle Scholar
  51. 51.
    Yong, C., Miikkulainen, R.: Coevolution of role-based cooperation in multi-agent systems. IEEE Transactions on Autonomous Mental Development (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Risto Miikkulainen
    • 1
  • Eliana Feasley
    • 1
  • Leif Johnson
    • 1
  • Igor Karpov
    • 1
  • Padmini Rajagopalan
    • 1
  • Aditya Rawal
    • 1
  • Wesley Tansey
    • 1
  1. 1.Department of Computer ScienceThe University of Texas at AustinAustinUSA

Personalised recommendations