Chess Neighborhoods, Function Combination, and Reinforcement Learning
Over the years, various research projects have attempted to develop a chess program that learns to play well given little prior knowledge beyond the rules of the game. Early on it was recognized that the key would be to adequately represent the relationships between the pieces and to evaluate the strengths or weaknesses of such relationships. As such, representations have developed, including a graph-based model. In this paper we extend the work on graph representation to a precise type of graph that we call a piece or square neighborhood. Specifically, a chessboard is represented as 64 neighborhoods, one for each square. Each neighborhood has a center, and 16 satellites corresponding to the pieces that are immediately close on the 4 diagonals, 2 ranks, 2 files, and 8 knight moves related to the square. Games are played and training values for boards are developed using temporal difference learning, as in other reinforcement learning systems. We then use a 2-layer regression network to learn. At the lower level the values (expected probability of winning) of the neighborhoods are learned and at the top they are combined based on their product and entropy. We report on relevant experiments including a learning experience on the Internet Chess Club (ICC) from which we can estimate a rating for the new program. The level of chess play achieved in a few days of training is comparable to a few months of work on previous systems such as Morph which is described as “one of the best from-scratch game learning systems, perhaps the best” .
Keywordslinear regression value function approximation temporal difference learning reinforcement learning computer chess exponentiated gradient gradient descent multi-layer neural nets
Unable to display preview. Download preview PDF.
- 1.Allen, J., Hamilton, E., and Levinson, R. New Advances in Adaptive Pattern-Oriented Chess (1997). In H.J. van den Herik and J.W.H., Uiterwijk. Advances in Computer Chess 8, pp. 312-233., Universiteit Maastricht, The Netherlands.Google Scholar
- 2.Baxter, J., Tridgell, A., and Weaver, L. A chess program that learns by combining TD(λ) with game tree search. In Proceedings of the 15th International Conference on Machine Learning (ICML-98), pages 28–36. Madision, WI. 1998. Morgan Kaufmann.Google Scholar
- 3.Ballard, D. H. An Introduction to Natural Computation. Cambridge: MIT Press.Google Scholar
- 4.Beal, D. F., & Smith, M.C. (1994). Random Evaluation in Chess. ICCA Journal, Vol. 17, No. 1, pp. 3–9 (A).Google Scholar
- 5.Beal, D. F., & Smith, M.C. Learning Piece Values Using Temporal Differences. Journal of The International Computer Chess Association, September 1997.Google Scholar
- 6.Beal, D. F., & Smith, M.C. First results from using temporal difference learning in Shogi. In H. J. van den Herik and H. Iida, editors, Proceedings of the First International Conference on Computers and Games ( CG-98), volume 1558 of Lecture Notes in Computer Science, page 114, Tsukuba, Japan, 1998. Springer-Verlag.Google Scholar
- 7.Bishop, Christopher M. Neural Networks for Pattern Recognition, Oxford Univ. Press, 1998. ISBN0-19-853864-2.Google Scholar
- 9.Christensen, J. and Korf, R. (1986). A unified theory of heuristic evaluation functions and its applications to learning. Proceedings of AAAI-86 (pp. 148–152).Google Scholar
- 10.Fürnkranz, J., Machine learning in computer chess: The next generation. International Computer Chess Association Journal, 19(3):147–160, September (1996).Google Scholar
- 11.Gherrity, M. A Game-Learning Machine. Ph.D thesis. University of California, San Diego. San Diego, CA. 1993.Google Scholar
- 12.Helmbold, D. P., Kivinen, J., and Warmuth, M. K. (1996a), Worst-case loss bounds for sigmoided linear neurons, in “Advances in Neural Information Processing Systems 8,” MIT Press, Cambridge, MA.Google Scholar
- 13.Herik, H.J. van den. A New Research Scope. International Computer Chess Association Journal 21(4), 1998.Google Scholar
- 14.Kivinen, J. and Warmuth, M. K. Additive versus exponentiated gradient updates for linear prediction. Information and Computation. Vol. 2, pp. 285–318, 1998.Google Scholar
- 15.Levinson, R. A., and Snyder, R. (1991). Adaptive pattern-oriented chess. In L. Birnbaum and G. Collins (Eds.), Proceedings of the 8th International Workshop on Machine Learning, pp. 85–89, Morgan Kaufmann.Google Scholar
- 16.Levinson, R. A., and Snyder, R., “Distance: Towards the Unification of Chess Knowledge”, International Computer Chess Association Journal 16(3): 123–136, September 1993.Google Scholar
- 17.Levinson, R. A., and Weber, R. J. (2000). “Pattern-level Temporal Difference Learning, Data Fusion, and Chess”. In SPIE’s 14th Annual Conference on Aerospace/Defense Sensing and Controls: Sensor Fusion: Architectures, Algorithms, and Applications IV.Google Scholar
- 19.Pearl, J. (1984). Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison-Wesley, Reading, Massachusetts.Google Scholar
- 20.Pellen, Luke. Neural net chess program Octavius: http://home.seol.net.au/luke/Octavius (1999).
- 22.Scott, J. Machine Learning in Games: the Morph Project, Swarthmore College, Swarthmore, PA. http://forum.swarthmore.edu/~jay/learn-game/projects/morph.html.
- 23.Slate, D.J., A chess program that uses its transposition table to learn from experience. International Computer Chess Association Journal 10(2):59–71, 1987.Google Scholar
- 24.Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9–44.Google Scholar
- 25.Sutton, R. S., & Barto, A.G. (1998). Reinforcement Learning: An Introduction. Cambridge: MIT Press.Google Scholar
- 26.Tesauro, G. Temporal Difference Learning and TD-Gammon. Communications of the ACM, Vol 38, No 3, March 1995.Google Scholar
- 28.Thrun, S., 1995. Learning to Play the Game of Chess. In Advances in Neural Information Processing Systems (NIPS) 7, G. Tesauro, D. Touretzky, and T. Leen (eds.), MIT Press.Google Scholar