Countering Evolutionary Forgetting in No-Limit Texas Hold’em Poker Agents

Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 399)

Abstract

No-Limit Texas Hold’em Poker is a stochastic game of imperfect information. Each player receives cards dealt randomly and does not know which cards his opponents have been dealt. These simple features result in No-Limit Texas Hold’em Poker having a large decision space in comparison to other classic games such as Backgammon and Chess. Evolutionary algorithms and neural networks have been shown to find solutions in large and non-linear decision spaces and have proven to aid decision making in No-Limit Texas Hold’em Poker. In this paper, a hybrid method known as evolving neural networks is used by No-Limit Texas Hold’em Poker playing agents to make betting decisions. When selecting a new generation of agents, evolutionary forgetting can result in selecting an agent with betting behaviour that has previously been shown to be inferior. To prevent this from occurring, we utilize two heuristics: halls of fame and co-evolution. In addition, we evaluate agent fitness using three fitness functions based upon, respectively, the length of time an agent survives in a tournament, the number of hands won in a tournament, and the average amount of money won across all hands in a tournament. Results show that the length of time an agent survives is indeed an appropriate measure of fitness. Results also show that utilizing halls of fame and co-evolution serve to further improve the fitness of agents. Finally, through monitoring the evolutionary progress of agents, we find that the skill level of agents improves when using our evolutionary heuristics.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Barone, L., While, L.: An adaptive learning model for simplified poker using evolutionary algorithms. In: Proceedings of the Congress on Evolutionary Computation, vol. 1, pp. 153–160 (1999)Google Scholar
  2. 2.
    Beattie, B., Nicolai, G., Gerhard, D., Hilderman, R.: Pattern classification in No-Limit Poker: A head start evolutionary approach. In: Canadian Conference on AI, pp. 204–215 (2007)Google Scholar
  3. 3.
    Billings, D., Papp, D., Pena, L., Schaeffer, J., Szafron, D.: Using selective-sampling simulations in poker. In: AAAI Spring Symposium on Search Techniches for Problem Solving Under Uncertainty and Incomplete Information (1999)Google Scholar
  4. 4.
    Billings, D., Davidson, A., Schaeffer, J., Szafron, D.: The challenge of poker. Artificial Intelligence 134, 201–240 (2002)MATHCrossRefGoogle Scholar
  5. 5.
    Billings, D., Burch, N., Davidson, A., Holte, R., Schaeffer, J., Schauenberg, T., Szafron, D.: Approximating game-theoretic optimal strategies for full-scale poker. In: Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (2003)Google Scholar
  6. 6.
    Billings, D.: Algorithms and Assessment in Computer Poker. PHD Dissertation. University of Alberta (2006)Google Scholar
  7. 7.
    Booker, L.: A No Limit Texas Hold’em poker playing agent. Master’s Thesis. University of London (2004)Google Scholar
  8. 8.
    Campbell, M., Hoane, A., Hsu, F.: Deep Blue. Artificial Intelligence 134, 57–83 (2002)MATHCrossRefGoogle Scholar
  9. 9.
    Donninger, C., Lorenz, U.: The Hydra project Xcell. Journal 53, 94–97 (2005)Google Scholar
  10. 10.
    Hauptman, A., Sipper, M.: GP-EndChess: Using genetic programming to evolve chess endgame players, booktitle. In: Proceedings of the 8th European Conference on Genetic Programming (2005)Google Scholar
  11. 11.
    Johanson, M.: Robust strategies and counter-strategies: Building a champion level computer poker player. Master’s thesis. University of Alberta (2007)Google Scholar
  12. 12.
    Kendall, G., Whitwell, G.: An evolutionary approach for tuning of a chess evaluation function using population dynamics. In: Proceedings of the 2001 IEEE Congress on Evolutionary Computation, pp. 995–1002 (2001)Google Scholar
  13. 13.
    Lubberts, A., Miikkulainen, R.: Co-evolving a go-playing neural network. In: Proceedings of the GECCO-01 Workshop on Coevolution: Turning Adaptive Algorithms Upon Themselves (2001)Google Scholar
  14. 14.
    Pollack, J., Blair, A.: Co-evolution in the successful learning of backgammon strategy. Machine Learning 32, 225–240 (1998)MATHCrossRefGoogle Scholar
  15. 15.
    Rosin, C.: Coevolutionary search among adversaries. PHD Dissertation. University fo California, San Diego (1997)Google Scholar
  16. 16.
    Samuel, A.L.: Some studies in machine learning using the game of checkers. IBM Journal of Research and Development (1959)Google Scholar
  17. 17.
    Schaeffer, J., Cublerson, J., Treloar, N., Knight, B., Lu, P., Szafron, D.: A word championship caliber checkers program. Artificial Intelligence 53, 273–289 (1992)CrossRefGoogle Scholar
  18. 18.
    Schauenberg, T.: Opponent modelling and search in poker. Master’s thesis. University of Alberta (2006)Google Scholar
  19. 19.
    Tesauro, G.: Programming backgammon using self-teaching neural nets. Artificial Intelligence 134, 181–199 (2002)MATHCrossRefGoogle Scholar
  20. 20.
    Thrun, S.: Learning to play the game of chess. Advances in Neural Information Processing Systems 7, 1069–1076 (1995)Google Scholar

Copyright information

© Springer-Verlag GmbH Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Dalhousie UniversityHalifaxCanada
  2. 2.University of ReginaReginaCanada

Personalised recommendations