Abstract
Generalization performance of learning agents depends on the training experience to which they have been exposed. In game-playing domains, that experience is determined by the opponents faced during learning. This analytical study investigates two characteristics of opponents in competitive coevolutionary learning: behavioral diversity and difficulty (performance against other players). To assess diversity, we propose a generic intra-game behavioral distance measure, that could be adopted to other sequential decision problems. We monitor both characteristics in two-population coevolutionary learning of Othello strategies, attempting to explain their relationship with the generalization performance achieved by the evolved solutions. The main observation is the existence of a non-obvious trade-off between difficulty and diversity, with the latter being essential for obtaining high generalization performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Popovici, E., Bucci, A., Wiegand, R.P., de Jong, E.D.: Coevolutionary principles. In: Rozenberg, G., Bäck, T., Kok, J.N. (eds.) Handbook of Natural Computing, pp. 987–1033. Springer, Heidelberg (2012)
Nolfi, S., Floreano, D.: Coevolving predator and prey robots: do “arms races” arise in artificial evolution? Artif. Life 4(4), 311–335 (1998)
Ficici, S.G., Pollack, J.B.: Challenges in coevolutionary learning: arms-race dynamics, open-endedness, and medicocre stable states. In: Proceedings of the Sixth International Conference on Artificial Life. ALIFE, pp. 238–247. MIT Press, Cambridge (1998)
Watson, R.A., Pollack, J.B.: Coevolutionary dynamics in a minimal substrate. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2001), pp. 702–709. Morgan Kaufmann (2001)
Jaśkowski, W., Liskowski, P., Szubert, M.G., Krawiec, K.: Improving coevolution by random sampling. In: Proceeding of the Fifteenth Annual Conference on Genetic and Evolutionary Computation Conference. GECCO 2013, pp. 1141–1148. ACM, New York (2013)
Chong, S.Y., Tino, P., Ku, D.C., Yao, X.: Improving generalization performance in co-evolutionary learning. IEEE Trans. Evol. Comput. 16(1), 70–85 (2012)
Goldberg, D.E., Richardson, J.: Genetic algorithms with sharing for multimodal function optimization. In: Proceedings of the Second International Conference on Genetic Algorithms and Their Application, pp. 41–49. L. Erlbaum Associates Inc., Hillsdale (1987)
Mahfoud, S.W.: Niching methods for genetic algorithms. Ph.D. thesis, University of Illinois at Urbana-Champaign, Urbana, IL (1995)
Sareni, B., Krahenbuhl, L.: Fitness sharing and niching methods revisited. IEEE Trans. Evol. Comput. 2(3), 97–106 (1998)
Rosin, C.D., Belew, R.K.: New methods for competitive coevolution. Evol. Comput. 5(1), 1–29 (1997)
Cartlidge, J., Bullock, S.: Combating coevolutionary disengagement by reducing parasite virulence. Evol. Comput. 12(2), 193–222 (2004)
Chong, S.Y., Tino, P., Yao, X.: Relationship between generalization and diversity in coevolutionary learning. IEEE Trans. Comput. Intell. AI Game. 1(3), 214–232 (2009)
Lehman, J., Stanley, K.O.: Abandoning objectives: evolution through the search for novelty alone. Evol. Comput. 19(2), 189–223 (2011)
Mouret, J.B., Doncieux, S.: Encouraging behavioral diversity in evolutionary robotics: an empirical study. Evol. Comput. 20(1), 91–133 (2012)
Gomez, F.J.: Sustaining diversity using behavioral information distance. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, GECCO 2009, pp. 113–120. ACM, New York (2009)
Gomes, J., Mariano, P., Christensen, A.L.: Novelty Search in competitive coevolution. In: Bartz-Beielstein, T., Branke, J., Filipič, B., Smith, J. (eds.) PPSN 2014. LNCS, vol. 8672, pp. 233–242. Springer, Heidelberg (2014)
Moriarty, D.E., Schultz, A.C., Grefenstette, J.J.: Evolutionary algorithms for reinforcement learning. J. Artif. Intell. Res. 11, 241–276 (1999)
Lucas, S.M., Runarsson, T.P.: Temporal difference learning versus co-evolution for acquiring othello position evaluation. In: Louis, S.J., Kendall, G. (eds.) Proceedings of the 2006 IEEE Symposium on Computational Intelligence and Games, CIG 2006, IEEE, pp. 52–59 (2006)
de Jong, E.D.: The MaxSolve algorithm for coevolution. In: Proceedings of the 2005 Conference on Genetic and Evolutionary Computation. GECCO 2005, pp. 483–489. ACM, New York (2005)
Beyer, H.G., Schwefel, H.P.: Evolution strategies-a comprehensive introduction. Nat. Comput. 1(1), 3–52 (2002)
de Jong, E.D., Pollack, J.B.: Ideal evaluation from coevolution. Evol. Comput. 12(2), 159–192 (2004)
Szubert, M.G., Jaśkowski, W., Liskowski, P., Krawiec, K.: Shaping fitness function for evolutionary learning of game strategies. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation. GECCO 2013, pp. 1149–1156. ACM, New York (2013)
Acknowledgments
This work has been supported by the Polish Ministry of Science and Higher Education, grant no. 09/91/DSMK/0568. W. Jaśkowski has been supported by the Polish National Science Centre grant no. DEC-2013/09/D/ST6/03932.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Szubert, M., Jaśkowski, W., Liskowski, P., Krawiec, K. (2015). The Role of Behavioral Diversity and Difficulty of Opponents in Coevolving Game-Playing Agents. In: Mora, A., Squillero, G. (eds) Applications of Evolutionary Computation. EvoApplications 2015. Lecture Notes in Computer Science(), vol 9028. Springer, Cham. https://doi.org/10.1007/978-3-319-16549-3_32
Download citation
DOI: https://doi.org/10.1007/978-3-319-16549-3_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16548-6
Online ISBN: 978-3-319-16549-3
eBook Packages: Computer ScienceComputer Science (R0)