Abstract
Novelty search is an algorithm which proposes open-ended exploration of the search space by maximising behavioural novelty, removing the need for an objective fitness function. However, we show that when applied to complex tasks, training through novelty alone is not sufficient to produce useful controllers. Alongside this, the definition of phenotypic behaviour significantly effects the strategies of the evolved solutions. Controller networks for the spaceship in the arcade game Asteroids were evolved with five different phenotypic distance measures. Each of these phenotypic measures are shown to produce controllers which adopt different strategies of play than controllers trained through standard objective fitness. Combined phenotypic novelty and objective fitness is also shown to produce differing strategies within the same evolutionary run. Our results demonstrate that for domains such as video games, where a diverse range of interesting behaviours are required, training agents through a combination of phenotypic novelty and objective fitness is a viable method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
It may be noted that with this particular phenotypic definition, the behaviour is measured hypothetically, before the task has been performed. Therefore when evaluating through novelty search alone (\(\lambda = 1.0\)), the game does not need to even be played.
References
Doncieux, S., Mouret, J.-B., Bredeche, N., Padois, V.: Evolutionary robotics: exploring new horizons. In: Doncieux, S., Bredèche, N., Mouret, J.-B. (eds.) New Horizons in Evolutionary Robotics. SCI, vol. 341, pp. 3–25. Springer, Heidelberg (2011)
Mouret, J.-B., Doncieux, S.: Encouraging behavioral diversity in evolutionary robotics: an empirical study. Evol. Comput. 20(1), 91–133 (2012)
Cully, A., Mouret, J.-B.: Behavioral repertoire learning in robotics. In: Proceeding of the Fifteenth Annual Conference on Genetic and Evolutionary Computation Conference, pp. 175–182. ACM (2013)
Lehman, J., Stanley, K.O., Miikkulainen, R.: Effective diversity maintenance in deceptive domains (2013)
Martin, P.R.: Measuring Behaviour: An Introductory Guide. Cambridge University Press, Cambridge (1993)
Lehman, J., Stanley, K.O.: Exploiting open-endedness to solve problems through the search for novelty. In: ALIFE, pp. 329–336 (2008)
Lehman, J., Stanley, K.O.: Abandoning objectives: evolution through the search for novelty alone. Evol. Comput. 19(2), 189–223 (2011)
Ruiz-Mirazo, K., Peretó, J., Moreno, A.: A universal definition of life: autonomy and open-ended evolution. Orig. Life Evol. Biosph. 34(3), 323–346 (2004)
Johannsen, W.: The genotype conception of heredity. Am. Nat. 45(531), 129–159 (1911)
Gomez, F.J.: Sustaining diversity using behavioral information distance. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 113–120. ACM (2009)
Cuccu, G., Gomez, F.: When novelty is not enough. In: Di Chio, C., et al. (eds.) EvoApplications 2011, Part I. LNCS, vol. 6624, pp. 234–243. Springer, Heidelberg (2011)
Mouret, J.-B.: Novelty-based multiobjectivization. In: Doncieux, S., Bredèche, N., Mouret, J.-B. (eds.) New Horizons in Evolutionary Robotics. SCI, vol. 341, pp. 139–154. Springer, Heidelberg (2011)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)
Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the nero video game. IEEE Trans. Evol. Comput. 9(6), 653–668 (2005)
Hastings, E.J., Guha, R.K., Stanley, K.O.: Evolving content in the galactic arms race video game. In: IEEE Symposium on Computational Intelligence and Games, CIG 2009, pp. 241–248. IEEE (2009)
Hausknecht, M., Lehman, J., Miikkulainen, R., Stone, P.: A neuroevolution approach to general atari game playing (2013)
Cardamone, L., Loiacono, D., Lanzi, P.L.: Evolving competitive car controllers for racing games with neuroevolution. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 1179–1186. ACM (2009)
Thawonmas, R., Ashida, T.: Evolution strategy for optimizing parameters in ms pac-man controller ice pambush 3. In: 2010 IEEE Symposium on Computational Intelligence and Games (CIG), pp. 235–240. IEEE (2010)
Hausknecht, M., Khandelwal, P., Miikkulainen, R., Stone, P.: Hyperneat-ggp: a hyperneat-based atari general game player. In: Proceedings of the Fourteenth International Conference on Genetic and Evolutionary Computation Conference, pp. 217–224. ACM (2012)
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Persson, M.: Minecraft (2009)
Murray, S., Ream, D., Doyle, R.: No man’s sky (2015)
Acknowledgements
We would like to thank the reviewers for their insightful comments and suggestions.
This work was funded by EPSRC through the Media and Arts Technology Programme, an RCUK Doctoral Training Centre EP/G03723X/1. Computational facilities were provided by the MidPlus Regional Centre of Excellence for Computational Science, Engineering and Mathematics, under EPSRC grant EP/K000128/1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Smith, D., Tokarchuk, L., Fernando, C. (2015). Evolving Diverse Strategies Through Combined Phenotypic Novelty and Objective Function Search. In: Mora, A., Squillero, G. (eds) Applications of Evolutionary Computation. EvoApplications 2015. Lecture Notes in Computer Science(), vol 9028. Springer, Cham. https://doi.org/10.1007/978-3-319-16549-3_28
Download citation
DOI: https://doi.org/10.1007/978-3-319-16549-3_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16548-6
Online ISBN: 978-3-319-16549-3
eBook Packages: Computer ScienceComputer Science (R0)