Evolving Game-Specific UCB Alternatives for General Video Game Playing

  • Ivan Bravi
  • Ahmed Khalifa
  • Christoffer Holmgård
  • Julian Togelius
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10199)


At the core of the most popular version of the Monte Carlo Tree Search (MCTS) algorithm is the UCB1 (Upper Confidence Bound) equation. This equation decides which node to explore next, and therefore shapes the behavior of the search process. If the UCB1 equation is replaced with another equation, the behavior of the MCTS algorithm changes, which might increase its performance on certain problems (and decrease it on others). In this paper, we use genetic programming to evolve replacements to the UCB1 equation targeted at playing individual games in the General Video Game AI (GVGAI) Framework. Each equation is evolved to maximize playing strength in a single game, but is then also tested on all other games in our test set. For every game included in the experiments, we found a UCB replacement that performs significantly better than standard UCB1. Additionally, evolved UCB replacements also tend to improve performance in some GVGAI games for which they are not evolved, showing that improvements generalize across games to clusters of games with similar game mechanics or algorithmic performance. Such an evolved portfolio of UCB variations could be useful for a hyper-heuristic game-playing agent, allowing it to select the most appropriate heuristics for particular games or problems in general.


General AI Genetic programming Monte-Carlo Tree Search 


  1. 1.
    Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)CrossRefzbMATHGoogle Scholar
  2. 2.
    Bäck, T., Schwefel, H.P.: An overview of evolutionary algorithms for parameter optimization. Evol. Comput. 1(1), 1–23 (1993)CrossRefGoogle Scholar
  3. 3.
    Baier, H., Winands, M.H.: Monte-Carlo Tree Search and minimax hybrids. In: 2013 IEEE Conference on Computational Intelligence in Games (CIG), pp. 1–8. IEEE (2013)Google Scholar
  4. 4.
    Browne, C.: Towards MCTS for creative domains. In: Proceedings of the International Conference on Computational Creativity, Mexico City, Mexico, pp. 96–101 (2011)Google Scholar
  5. 5.
    Browne, C.B., Powley, E., Whitehouse, D., Lucas, S.M., Cowling, P.I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., Colton, S.: A survey of Monte Carlo Tree Search methods. IEEE Trans. Comput. Intell. AI Games 4(1), 1–43 (2012)CrossRefGoogle Scholar
  6. 6.
    Burke, E.K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., Qu, R.: Hyper-heuristics: a survey of the state of the art. J. Oper. Res. Soc. 64(12), 1695–1724 (2013)CrossRefGoogle Scholar
  7. 7.
    Cazenave, T.: Evolving Monte Carlo Tree Search Algorithms. Dept. Inf., Univ. Paris 8 (2007)Google Scholar
  8. 8.
    Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing, vol. 53. Springer, Heidelberg (2003)CrossRefzbMATHGoogle Scholar
  9. 9.
    Frydenberg, F., Andersen, K.R., Risi, S., Togelius, J.: Investigating MCTS modifications in general video game playing. In: 2015 IEEE Conference on Computational Intelligence and Games (CIG), pp. 107–113. IEEE (2015)Google Scholar
  10. 10.
    Jacobsen, E.J., Greve, R., Togelius, J.: Monte Mario: platforming with MCTS. In: Proceedings of the 2014 Conference on Genetic and Evolutionary Computation, pp. 293–300. ACM (2014)Google Scholar
  11. 11.
    Justesen, N., Mahlmann, T., Togelius, J.: Online evolution for multi-action adversarial games. In: Squillero, G., Burelli, P. (eds.) EvoApplications 2016. LNCS, vol. 9597, pp. 590–603. Springer, Cham (2016). doi: 10.1007/978-3-319-31204-0_38CrossRefGoogle Scholar
  12. 12.
    Kotthoff, L.: Algorithm selection for combinatorial search problems: a survey. AI Mag. 35(3), 48–60 (2014)CrossRefGoogle Scholar
  13. 13.
    Levine, J., Congdon, C.B., Ebner, M., Kendall, G., Lucas, S.M., Miikkulainen, R., Schaul, T., Thompson, T.: General video game playing. Dagstuhl Follow-Ups 6 (2013)Google Scholar
  14. 14.
    McGuinness, C.: Monte Carlo Tree Search: Analysis and Applications. Ph.D. thesis (2016)Google Scholar
  15. 15.
    Park, H., Kim, K.J.: MCTS with influence map for general video game playing. In: 2015 IEEE Conference on Computational Intelligence and Games (CIG), pp. 534–535. IEEE (2015)Google Scholar
  16. 16.
    Perez, D., Samothrakis, S., Lucas, S., Rohlfshagen, P.: Rolling horizon evolution versus tree search for navigation in single-player real-time games. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 351–358. ACM (2013)Google Scholar
  17. 17.
    Perez, D., Samothrakis, S., Togelius, J., Schaul, T., Lucas, S., Couëtoux, A., Lee, J., Lim, C.U., Thompson, T.: The 2014 General Video Game Playing Competition (2015)Google Scholar
  18. 18.
    Perez-Liebana, D., Samothrakis, S., Togelius, J., Schaul, T., Lucas, S.M.: General Video Game AI: Competition, Challenges and Opportunities (2016)Google Scholar
  19. 19.
    Pettit, J., Helmbold, D.: Evolutionary learning of policies for MCTS simulations. In: Proceedings of the International Conference on the Foundations of Digital Games, pp. 212–219. ACM (2012)Google Scholar
  20. 20.
    Poli, R., Langdon, W.B., McPhee, N.F., Koza, J.R.: A Field Guide to Genetic Programming., Raleigh (2008)Google Scholar
  21. 21.
    Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)CrossRefGoogle Scholar
  22. 22.
    Rimmel, A., Teytaud, O., Lee, C.S., Yen, S.J., Wang, M.H., Tsai, S.R.: Current frontiers in computer go. IEEE Trans. Comput. Intell. AI Games 2(4), 229–238 (2010)CrossRefGoogle Scholar
  23. 23.
    Whitley, D.: A genetic algorithm tutorial. Stat. Comput. 4(2), 65–85 (1994)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Ivan Bravi
    • 1
  • Ahmed Khalifa
    • 2
  • Christoffer Holmgård
    • 2
  • Julian Togelius
    • 2
  1. 1.Dipartimento di Elettronica, Informatica e BioingegneriaPolitecnico di MilanoMilanoItaly
  2. 2.Tandon School of EngineeringNew York UniversityNew York CityUSA

Personalised recommendations