Abstract
DeepMind’s recent spectacular success in using deep convolutional neural nets and machine learning to build superhuman level agents—e.g. for Atari games via deep Q-learning and for the game of Go via other deep Reinforcement Learning methods—raises many questions, including to what extent these methods will succeed in other domains. In this paper we consider DQL for the game of Hex: after supervised initializing, we use self-play to train NeuroHex, an 11-layer convolutional neural network that plays Hex on the 13 \(\times \) 13 board. Hex is the classic two-player alternate-turn stone placement game played on a rhombus of hexagonal cells in which the winner is whomever connects their two opposing sides. Despite the large action and state space, our system trains a Q-network capable of strong play with no search. After two weeks of Q-learning, NeuroHex achieves respective win-rates of 20.4% as first player and 2.1% as second player against a 1-s/move version of MoHex, the current ICGA Olympiad Hex champion. Our data suggests further improvement might be possible with more training time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anshelevich, V.V.: The game of Hex: an automatic theorem proving approach to game programming. In: AAAI/IAAI, pp. 189–194 (2000)
Arneson, B., Hayward, R., Henderson, P.: Wolve wins Hex tournament. ICGA J. 32, 49–53 (2008)
Arneson, B., Hayward, R.B., Henderson, P.: Monte Carlo tree search in Hex. IEEE Trans. Comput. Intell. AI Games 2(4), 251–258 (2010)
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I.J., Bergeron, A., Bouchard, N., Bengio, Y.: Theano: new features and speed improvements. In: NIPS 2012 Deep Learning and Unsupervised Feature Learning Workshop (2012)
Baudiš, P., Gailly, J.: PACHI: state of the art open source go program. In: Herik, H.J., Plaat, A. (eds.) ACG 2011. LNCS, vol. 7168, pp. 24–38. Springer, Heidelberg (2012). doi:10.1007/978-3-642-31866-5_3
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., Bengio, Y.: Theano: a CPU and GPU math expression compiler. In: Proceedings of the Python for Scientific Computing Conference (SciPy) (2010). Oral Presentation
Gardner, M.: Mathematical games. Sci. Am. 197(1), 145–150 (1957)
Hayward, R.B.: MoHex wins Hex tournament. ICGA J. 36(3), 180–183 (2013)
Huang, S.-C., Arneson, B., Hayward, R.B., Müller, M., Pawlewicz, J.: MoHex 2.0: a pattern-based MCTS Hex player. In: Herik, H.J., Iida, H., Plaat, A. (eds.) CG 2013. LNCS, vol. 8427, pp. 60–71. Springer, Cham (2014). doi:10.1007/978-3-319-09165-5_6
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Reisch, S.: Hex ist PSPACE-vollständig. Acta Informatica 15, 167–191 (1981)
Shannon, C.E.: Computers and automata. Proc. Inst. Radio Eng. 41, 1234–1241 (1953)
Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., Hassabis, D.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Tesauro, G.: Temporal difference learning and TD-gammon. Commun. ACM 38(3), 58–68 (1995)
Tieleman, T., Hinton, G.: Lecture 6.5–RmsProp: divide the gradient by a running average of its recent magnitude. COURSERA Neural Netw. Mach. Learn. (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Young, K., Vasan, G., Hayward, R. (2017). NeuroHex: A Deep Q-learning Hex Agent. In: Cazenave, T., Winands, M., Edelkamp, S., Schiffel, S., Thielscher, M., Togelius, J. (eds) Computer Games. CGW GIGA 2016 2016. Communications in Computer and Information Science, vol 705. Springer, Cham. https://doi.org/10.1007/978-3-319-57969-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-57969-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-57968-9
Online ISBN: 978-3-319-57969-6
eBook Packages: Computer ScienceComputer Science (R0)