Online Evolution for Multi-action Adversarial Games

  • Niels Justesen
  • Tobias MahlmannEmail author
  • Julian Togelius
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9597)


We present Online Evolution, a novel method for playing turn-based multi-action adversarial games. Such games, which include most strategy games, have extremely high branching factors due to each turn having multiple actions. In Online Evolution, an evolutionary algorithm is used to evolve the combination of atomic actions that make up a single move, with a state evaluation function used for fitness. We implement Online Evolution for the turn-based multi-action game Hero Academy and compare it with a standard Monte Carlo Tree Search implementation as well as two types of greedy algorithms. Online Evolution is shown to outperform these methods by a large margin. This shows that evolutionary planning on the level of a single move can be very effective for this sort of problems.


Action Sequence Time Budget Game State Strategy Game Spell Action 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Branavan, S., Silver, D., Barzilay, R.: Non-linear monte-carlo search in civilization ii. In: AAAI Press/International Joint Conferences on Artificial Intelligence (2011)Google Scholar
  2. 2.
    Browne, C.B., Powley, E., Whitehouse, D., Lucas, S.M., Cowling, P., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., Colton, S., et al.: A survey of monte carlo tree search methods. IEEE Trans. Comput. Intell. AI Games 4(1), 1–43 (2012)CrossRefGoogle Scholar
  3. 3.
    Cardamone, L., Loiacono, D., Lanzi, P.L.: Evolving competitive car controllers for racing games with neuroevolution. In: Proceedings of the 11th Annual conference on Genetic and evolutionary computation, pp. 1179–1186. ACM (2009)Google Scholar
  4. 4.
    Chaslot, G., Bakkes, S., Szita, I., Spronck, P.: Monte-carlo tree search: a new framework for game ai. In: AIIDE (2008)Google Scholar
  5. 5.
    Churchill, D., Buro, M.: Portfolio greedy search and simulation for large-scale combat in starcraft. In: 2013 IEEE Conference on Computational Intelligence in Games (CIG), pp. 1–8. IEEE (2013)Google Scholar
  6. 6.
    Elias, G.S., Garfield, R., Gutschera, K.R.: Characteristics of Games. MIT Press, Cambridge (2012)Google Scholar
  7. 7.
    Gelly, S., Wang, Y.: Exploration exploitation in go: uct for monte-carlo go. In: NIPS: Neural Information Processing Systems Conference On-line trading of Exploration and Exploitation Workshop (2006)Google Scholar
  8. 8.
    Glover, F., Laguna, M.: Tabu Search*. Springer, New York (2013)zbMATHGoogle Scholar
  9. 9.
    Helmbold, D.P., Parker-Wood, A.: All-moves-as-first heuristics in monte-carlo go. In: IC-AI, pp. 605–610 (2009)Google Scholar
  10. 10.
    Justesen, N.: Artificial intelligence for hero academy. Master’s thesis, IT University of Copenhagen (2015)Google Scholar
  11. 11.
    Justesen, N., Tillman, B., Togelius, J., Risi, S.: Script-and cluster-based uct for starcraft. In: 2014 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1–8. IEEE (2014)Google Scholar
  12. 12.
    Kozelek, T.: Methods of mcts and the game arimaa. Charles University, Prague, Faculty of Mathematics and Physics (2009)Google Scholar
  13. 13.
    Levine, J., Congdon, C.B., Ebner, M., Kendall, G., Lucas, S.M., Miikkulainen, R., Schaul, T., Thompson, T., Lucas, S.M., Mateas, M., et al.: General video game playing. Artif. Comput. Intell. Games 6, 77–83 (2013)Google Scholar
  14. 14.
    Mahfoud, S.W.: Niching methods for genetic algorithms. Urbana 51(95001), 62–94 (1995)Google Scholar
  15. 15.
    Neumann, J.V.: Zur Theorie der Gesellschaftsspiele. Math. Ann. 100(1), 295–320 (1928)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Perez, D., Rohlfshagen, P., Lucas, S.M.: Monte-Carlo tree search for the physical travelling salesman problem. In: Di Chio, C., et al. (eds.) EvoApplications 2012. LNCS, vol. 7248, pp. 255–264. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  17. 17.
    Perez, D., Samothrakis, S., Lucas, S., Rohlfshagen, P.: Rolling horizon evolution versus tree search for navigation in single-player real-time games. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 351–358. ACM (2013)Google Scholar
  18. 18.
    Rosin, C.D., Belew, R.K.: New methods for competitive coevolution. Evol. Comput. 5(1), 1–29 (1997)CrossRefGoogle Scholar
  19. 19.
    Shannon, C.E.: XXII. programming a computer for playing chess. Lond. Edinb. Dublin Philos. Mag. J. Sci. 41(314), 256–275 (1950)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Togelius, J., Karakovskiy, S., Baumgarten, R.: The 2009 mario ai competition. In: 2010 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2010)Google Scholar
  21. 21.
    Togelius, J., Karakovskiy, S., Koutník, J., Schmidhuber, J.: Super mario evolution. In: IEEE Symposium on Computational Intelligence and Games, 2009, CIG 2009, pp. 156–161. IEEE (2009)Google Scholar
  22. 22.
    Zhou, A., Qu, B.Y., Li, H., Zhao, S.Z., Suganthan, P.N., Zhang, Q.: Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol. Comput. 1(1), 32–49 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Niels Justesen
    • 1
  • Tobias Mahlmann
    • 2
    Email author
  • Julian Togelius
    • 3
  1. 1.IT University of CopenhagenCopenhagenDenmark
  2. 2.Lund UniversityLundSweden
  3. 3.New York UniversityNew YorkUSA

Personalised recommendations