Advertisement

Learning to Trade in Strategic Board Games

  • Heriberto Cuayáhuitl
  • Simon Keizer
  • Oliver Lemon
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 614)

Abstract

Automated agents in multiplayer board games often need to trade resources with their opponents—and trading strategically can lead to higher winning rates. While rule-based agents can be used for such a purpose, here we opt for a data-driven approach based on examples from human players for automatic trading in the game “Settlers of Catan”. Our experiments are based on data collected from human players trading in text-based Natural Language. We compare the performance of Bayesian Networks, Conditional Random Fields, and Random Forests in the task of ranking trading offers, and evaluate them both in an offline setting and online while playing the game against a rule-based baseline. Experimental results show that agents trained from data from average human players can outperform rule-based trading behavior, and that the Random Forest model achieves the best results.

Keywords

Bayesian Network Random Forest Statistical Classifier Board Game Random Forest Classifier 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

Funding from the European Research Council (ERC) project “STAC: Strategic Conversation” no. 269427 is gratefully acknowledged. See http://www.irit.fr/STAC/. We would like to thank the following members of the STAC project for helpful discussions: Markus Guhe, Eric Kow, Mihai Dobre, Ioannis Efstathiou, Verena Rieser, Alex Lascarides, and Nicholas Asher.

References

  1. 1.
    Afantenos, S., Asher, N., Benamara, F., Cadilhac, A., Dégremont, C., Denis, P., Guhe, M., Keizer, S., Lascarides, A., Lemon, O., Muller, P., Paul, S., Rieser, V., Vieu, L.: Developing a corpus of strategic conversation in the Settlers of Catan. In: Workshop on the Semantics and Pragmatics of Dialogue SeineDial, Paris, France (2012). https://hal.inria.fr/hal-00750618
  2. 2.
    Asher, N., Lascarides, A.: Commitments, beliefs and intentions in dialogue. In: Proceedings of SemDial, pp. 35–42 (2008)Google Scholar
  3. 3.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Choi, J., Park, J., Park, H., Park, J.I.: iHand: an interactive bare-hand-based augmented reality interface on commercial mobile phones. Opt. Eng. 52(2), 027206 (2013)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Cooper, G., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9(4), 309–347 (1992)zbMATHGoogle Scholar
  6. 6.
    Cozman, F.G.: Generalizing variable elimination in Bayesian networks. In: Workshop on Probabilistic Reasoning in Artificial Intelligence, pp. 27–32 (2000)Google Scholar
  7. 7.
    Criminisi, A., Shotton, J., Konukoglu, E.: Decision forests: a unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning. Found. Trends Comput. Graph. Vis. 7(2–3), 81–227 (2012)zbMATHGoogle Scholar
  8. 8.
    Cuayáhuitl, H., van Otterlo, M., Dethlefs, N., Frommberger, L.: Machine learning for interactive systems and robots: a brief introduction. In: 2nd Workshop on Machine Learning for Interactive Systems (MLIS), pp. 19–28. ACM (2013)Google Scholar
  9. 9.
    Dethlefs, N., Cuayáhuitl, H.: Hierarchical reinforcement learning for situated natural language generation. Nat. Lang. Eng. 21(3), 391–435 (2015)CrossRefGoogle Scholar
  10. 10.
    Efstathiou, I., Lemon, O.: Learning to manage risk in non-cooperative dialogues. In: Proceedings of SEMDIAL (2014)Google Scholar
  11. 11.
    Efstathiou, I., Lemon, O.: Learning non-cooperative dialogue behaviours. In: SIGDIAL (2014)Google Scholar
  12. 12.
    Efstathiou, I., Lemon, O.: Learning non-cooperative dialogue policies to beat opponent models: ‘the good, the bad and the ugly’. In: Proceedings of SEMDIAL (2015)Google Scholar
  13. 13.
    Fürnkranz, J.: Machine learning in games: a survey. In: Machines that Learn to Play Games, Chapter 2, pp. 11–59. Nova Science Publishers (2000)Google Scholar
  14. 14.
    Georgila, K., Nelson, C., Traum, D.: Single-agent vs. multi-agent techniques for concurrent reinforcement learning of negotiation dialogue policies. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, USA, pp. 500–510, September 2014Google Scholar
  15. 15.
    Georgila, K., Traum, D.: Reinforcement learning of argumentation dialogue policies in negotiation. In: Proceedings of INTERSPEECH (2011)Google Scholar
  16. 16.
    Guhe, M., Lascarides, A.: Game strategies for the Settlers of Catan. In: 2014 IEEE Conference on Computational Intelligence and Games, CIG 2014, Dortmund, Germany, 26–29 August 2014, pp. 1–8 (2014)Google Scholar
  17. 17.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2nd edn. Springer, New York (2009)CrossRefzbMATHGoogle Scholar
  18. 18.
    Hiraoka, T., Georgila, K., Nouri, E., Traum, D., Nakamura, S.: Reinforcement learning in multi-party trading dialog. In: Proceedings of the SIGDIAL 2015 Conference, Prague, Czech Republic, pp. 32–41, September 2015Google Scholar
  19. 19.
    Kudo, T.: CRF++: Yet another CRF toolkit (2005). http://crfpp.sourceforge.net
  20. 20.
    Lemon, O.: Adaptive natural language generation in dialogue using reinforcement learning. In: Proceedings of SEMDIAL (2008)Google Scholar
  21. 21.
    Maddison, C.J., Huang, A., Sutskever, I., Silver, D.: Move evaluation in go using deep convolutional neural networks. CoRR abs/1412.6564 (2014)Google Scholar
  22. 22.
    McFarlin, M.: 10 great board games for traders. Futures Magazine (2013). http://www.futuresmag.com/2013/10/02/10-great-board-games-for-traders
  23. 23.
    Pfeiffer, M.: Reinforcement learning of strategies for Settlers of Catan. In: International Conference on Computer Games: Artificial Intelligence, Design and Education (2004)Google Scholar
  24. 24.
    Pietquin, O., Lopez, M.: Machine learning for interactive systems: challenges and future trends. In: Proceedings of the Workshop Affect, Compagnon Artificiel (WACAI) (2014)Google Scholar
  25. 25.
    Runarsson, T.P., Lucas, S.M.: Preference learning for move prediction and evaluation function approximation in Othello. IEEE Trans. Comput. Intell. AI Games 6(3), 300–313 (2014)CrossRefGoogle Scholar
  26. 26.
    Shim, J., Arkin, R.: A taxonomy of robot deception and its benefits in HRI. In: Proceedings of IEEE Systems, Man and Cybernetics Conference (2013)Google Scholar
  27. 27.
    Szita, I., Chaslot, G., Spronck, P.: Monte-Carlo tree search in Settlers of Catan. In: van den Herik, H.J., Spronck, P. (eds.) ACG 2009. LNCS, vol. 6048, pp. 21–32. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  28. 28.
    Tesauro, G.: Temporal difference learning and TD-Gammon. Commun. ACM 38(3), 58–68 (1995)CrossRefGoogle Scholar
  29. 29.
    Thomas, R., Hammond, K.J.: Java settlers: a research environment for studying multi-agent negotiation. In: Intelligent User Interfaces (IUI), p. 240 (2002)Google Scholar
  30. 30.
    Traum, D.: Extended abstract: computational models of non-cooperative dialogue. In: Proceedings of SIGdial Workshop on Discourse and Dialogue (2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Heriberto Cuayáhuitl
    • 1
  • Simon Keizer
    • 1
  • Oliver Lemon
    • 1
  1. 1.School of Mathematical and Computer SciencesHeriot-Watt UniversityEdinburghUK

Personalised recommendations