Abstract
Many games enjoyed by players primarily consist of a matching system that allows the player to cooperate or compete with other players with similar scores. However, the method of matching only the play score can easily lose interest because it does not consider the opponent’s playstyle or strategy. In this study, we propose a self-supervised contrastive learning framework that can enhance the understanding of game replay data to create a more sophisticated matching system. We use actor-critic-based reinforcement learning agents to collect many replay data. We define a positive pair and negative examples to perform contrastive learning. Positive pair is defined by sampling from the frames of the same replay data, otherwise negatives. To evaluate the performance of the proposed framework, we use Facebook ELF, a real-time strategy game, to collect replay data and extract data features from pre-trained neural networks. Furthermore, we apply k-means clustering with the extracted features to visually demonstrate that different play patterns and proficiencies can be clustered appropriately. We present our clustering results on replay data and show that the proposed framework understands the nature of the data with consecutive frames.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Yusfrizal, Adhar, D., Indriani, U., Panggabean, E., Sabir, A., Kurniawan, H.: Application of the fisher-yates shuffle algorithm in the game matching the world monument picture. In: 2020 2nd International Conference on Cybernetics and Intelligent System (ICORIS), pp. 1–6. IEEE, Manado, Indonesia (2020)
Vinyals, O., et al.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019)
Kurach, K., et al.: Google research football: a novel reinforcement learning environment. arXiv preprint arXiv:1907.11180 (2019)
Ravari, Y.N., Snader, B., Pieter, S.: Starcraft winner prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 12, no. 1 (2016)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Ren, S., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inform. Process. Syst. 28, 91–99 (2015)
Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
Oprea, S., et al.: A review on deep learning techniques for video prediction. IEEE Trans. Pattern Analysis Mach. Intell. 44(6), 2806–2826 (2020)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
Vinyals, O., et al.: Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017)
Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision (2015)
Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: European Conference on Computer Vision. Springer, Cham (2016)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018)
van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning. PMLR (2020)
Misra, I., van der Maaten, L.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
Liu, H., Abbeel, P.: Unsupervised active pre-training for reinforcement learning (2020)
Stooke, A., Lee, K., Abbeel, P., Laskin, M.: Decoupling representation learning from reinforcement learning. In: International Conference on Machine Learning. PMLR (2021)
Qian, R., Meng, T., Gong, B., Yang, M.-H., Wang, H., Belongie, S., Cui, Y.: Spatiotemporal contrastive video representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML’10), pp. 807–814. Omnipress, Madison, WI, USA (2010)
Silver, D., et al.: Deterministic policy gradient algorithms. In: International Conference on Machine Learning. PMLR (2014)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)
Acknowledgments
This research was supported by the Agency for Defense Development (UI2100062D).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lee, Y.J. et al. (2023). Self-supervised Contrastive Learning for Predicting Game Strategies. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2022. Lecture Notes in Networks and Systems, vol 542. Springer, Cham. https://doi.org/10.1007/978-3-031-16072-1_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-16072-1_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16071-4
Online ISBN: 978-3-031-16072-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)