Computational Experiments with the RAVE Heuristic
The Monte-Carlo tree search algorithm Upper Confidence bounds applied to Trees (UCT) has become extremely popular in computer games research. The Rapid Action Value Estimation (RAVE) heuristic is a strong estimator that often improves the performance of UCT-based algorithms. However, there are situations where RAVE misleads the search whereas pure UCT search can find the correct solution. Two games, the simple abstract game Sum of Switches (SOS) and the game of Go, are used to study the behavior of the RAVE heuristic. In SOS, RAVE updates are manipulated to mimic game situations where RAVE misleads the search. Such false RAVE updates are used to create RAVE overestimates and underestimates. A study of the distributions of mean and RAVE values reveals great differences between Go and SOS. While the RAVE-max update rule is able to correct extreme cases of RAVE underestimation, it is not effective in closer to practical settings and in Go.
Unable to display preview. Download preview PDF.
- 2.Gelly, S., Wang, Y., Munos, R., Teytaud, O.: Modification of UCT with patterns in Monte-Carlo Go, Technical Report RR-6062, INRIA, France (2006)Google Scholar
- 3.Finnsson, H., Björnsson, Y.: Simulation-based approach to General Game Playing. In: Fox, D., Gomes, C. (eds.) AAAI, pp. 259–264. AAAI Press, Menlo Park (2008)Google Scholar
- 6.Winands, M., Björnsson, Y.: Evaluation function based Monte-Carlo LOA. In: , pp. 33–44Google Scholar
- 7.Brügmann, B.: Monte Carlo Go (March 1993) (unpublished manuscript), http://www.cgl.ucsf.edu/go/Programs/Gobble.html
- 9.Tom, D., Müller, M.: A study of UCT and its enhancements in an artificial game. In: , pp. 55–64Google Scholar
- 10.Teytaud, F., Teytaud, O.: Creating an Upper-Confidence-Tree program for Havannah. In: , pp. 65–74Google Scholar
- 11.Enzenberger, M., Müller, M.: Fuego (2008), http://fuego.sf.net/ (Retrieved December 22, 2008)
- 12.Silver, D.: Reinforcement Learning and Simulation-Based Search. PhD thesis, University of Alberta (2009)Google Scholar
- 13.Tom, D.: Investigating UCT and RAVE: Steps Towards a More Robust Method. Master’s thesis, University of Alberta, Department of Computing Science (2010)Google Scholar
- 14.Enzenberger, M., Müller, M., Arneson, B., Segal, R.: Fuego – an open-source framework for board games and Go engine based on Monte-Carlo tree search. Submitted to IEEE Transactions on Computational Intelligence and AI in Games (2010)Google Scholar
- 15.van den Herik, H.J., Spronck, P. (eds.): ACG 2009. LNCS, vol. 6048. Springer, Heidelberg (2010)Google Scholar