Skip to main content

Advertisement

SpringerLink
Log in

Deep ensemble learning of tactics to control the main force in a real-time strategy game

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Professional StarCraft game players are likely to focus on the management of the most important group of units (called the main force) during gameplay. Although macro-level skills have been observed in human game replays, there has been little study of the high-level knowledge used for tactical decision-making, nor exploitation thereof to create AI modules. In this paper, we propose a novel tactical decision-making model that makes decisions to control the main force. We categorized the future movement direction of the main force into six classes (e.g., toward the enemy’s main base). The model learned to predict the next destination of the main force based on the large amount of experience represented in replays of human games. To obtain training data, we extracted information from 12,057 replay files produced by human players and obtained the position and movement direction of the main forces through a novel detection algorithm. We applied convolutional neural networks and a Vision Transformer to deal with the high-dimensional state representation and large state spaces. Furthermore, we analyzed human tactics relating to the main force. Model learning success rates of 88.5%, 76.8%, and 56.9% were achieved for the top-3, -2, and -1 accuracies, respectively. The results show that our method is capable of learning human macro-level intentions in real-time strategy games.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Adil K, Jiang F, Liu S, Jifara W, Tian Z, Fu Y (2017) State-of-the-art and open challenges in rts game-ai and starcraft. Int J Adv Comput Sci Appl 8(12):16–24

    Google Scholar 

  2. Agarwal R, Schuurmans D, Norouzi M (2020) An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp 104–114 

  3. Andersen PA, Goodwin M, Granmo OC (2018) Deep RTS: a game environment for deep reinforcement learning in real-time strategy games. In 2018 IEEE conference on computational intelligence and games (CIG), pp 1–8

  4. Baek IC, Kim KJ (2018, August) Web-Based Interface for Data Labeling in StarCraft. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp 1–2

  5. Barriga NA, Stanescu M, Besoain F, Buro M (2019) Improving rts game ai by supervised policy learning, tactical search, and deep reinforcement learning. IEEE Comput Intell Mag 14(3):8–18. https://doi.org/10.1109/MCI.2019.2919363

    Article  Google Scholar 

  6. Beal R, Chalkiadakis G, Norman TJ, Ramchurn SD (2020) Optimising game tactics for football. https://arxiv.org/pdf/2003.10294.pdf

  7. Burda Y, Edwards H, Storkey A,  Klimov O (2019) Exploration by random network distillation. In Seventh International Conference on Learning Representations, pp 1–17

  8. Buro M (2003) Real-time strategy gaines: a new AI research challenge. In Proceedings of the 18th international joint conference on Artificial intelligence, pp 1534–1535

  9. Buro M (2002) ORTS: A hack-free RTS game environment. In International Conference on Computers and Games, Berlin, Heidelberg: Springer Berlin Heidelberg, pp 280–291

  10. Čertický M, Churchill D, Kim K-J, Čertický M, Kelly R (2018) Star-craft ai competitions, bots, and tournament manager software. IEEE Trans Games 11(3):227–237

    Article  Google Scholar 

  11. Chen L, Lu K, Rajeswaran A, Lee K, Grover A, Laskin M, Abbeel P, Srinivas A, Mordatch I (2021) Decision transformer: Reinforcement learning via sequence modeling. Adv Neural Inform Process Syst 34:15084–15097

    Google Scholar 

  12. Cho HC, Kim KJ, Cho SB (2013) Replay-based strategy prediction and build order adaptation for StarCraft AI bots. In 2013 IEEE Conference on Computational Intelligence in Games (CIG) pp 1–7 

  13. Cho H, Park H, Kim C-Y, Kim K-J (2016) Investigation of the effect of “fog of war” in the prediction of starcraft strategy using machine learning. Comput Entertain (CIE) 14(1):1–16

    Article  Google Scholar 

  14. Churchill D, Buro M (2011) Build order optimization in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 7(1):14–19

    Google Scholar 

  15. Churchill D, Buro M (2012) Incorporating search algorithms into RTS game agents. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 8(3):2–7

    Google Scholar 

  16. Churchill D, Buro M (2013) Portfolio greedy search and simulation for large-scale combat in StarCraft. In 2013 IEEE Conference on Computational Intelligence in Games (CIG) pp 1–8

  17. Churchill D, Saffidine A, Buro M (2012) Fast heuristic search for RTS game combat scenarios. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment  8(1):112–117

  18. Ciolino M, Kalin J, Noever D (2020) The Go Transformer: Natural Language Modeling for Game Play. In 2020 Third International Conference on Artificial Intelligence for Industries (AI4I) pp 23–26

  19. Dereszynski E, Hostetler J, Fern A, Dietterich T, Hoang TT, Udarbe M (2011) Learning probabilistic behavior models in real-time strategy games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 7(1):20–25

  20. Dietterich TG (2000) Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21–23, 2000 Proceedings 1 pp 1–15 Springer Berlin Heidelberg

  21. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Houlsby N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations

  22. Farooq SS, Oh I-S, Kim M-J, Kim KJ (2016) Starcraft ai competition report. AI Mag 37(2):102–107

    Google Scholar 

  23. Foerster J, Nardelli N, Farquhar G, Afouras T, Torr PH, Kohli P, Whiteson S (2017) Stabilising experience replay for deep multi-agent reinforcement learning. In International conference on machine learning, PMLR pp 1146–1155

  24. Gabriel I, Negru V, Zaharie D (2012) Neuroevolution based multi-agent system for micromanagement in real-time strategy games. In Proceedings of the fifth balkan conference in informatics pp 32–39

  25. Garćıa-Sánchez P, Tonda A, Mora AM, Squillero G, Merelo JJ (2015) Towards automatic StarCraft strategy generation using genetic programming. In 2015 IEEE Conference on Computational Intelligence and Games (CIG) pp 284–291

  26. Goodfellow  IJ, Mirza M, Xu B, Ozair S, Courville A, Bengio Y (2014) Generative Adversarial Networks. https://arxiv.org/pdf/1406.2661.pdf

  27. Gudmundsson SF, Eisen P, Poromaa E, Nodet A, Purmonen S, Kozakowski B, Cao L (2018) Human-like playtesting with deep learning. In 2018 IEEE Conference on Computational Intelligence and Games (CIG) pp 1–8

  28. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition pp 770–778

  29. Ho J, Ermon S (2016) Generative adversarial imitation learning. Adv Neural Inf Process Systs 29

  30. Hoel C-J, Driggs-Campbell K, Wolff K, Laine L, Kochenderfer MJ (2019) Combining planning and deep reinforcement learning in tactical decision making for autonomous driving. IEEE transactions on intelligent vehicles 5(2):294–305

    Article  Google Scholar 

  31. Hu Y, Li J, Li X, Pan G, Xu M (2018) Knowledge-guided agent-tactic-aware learning for StarCraft micromanagement. In Proceedings of the 27th International Joint Conference on Artificial Intelligence pp 1471–1477

  32. Huang J, Yang W (2018) A multi-size convolution neural network for RTS games winner prediction. In MATEC Web of Conferences 232:01054. EDP Sciences

  33. Huang W, Yin Q, Zhang J, Huang K (2021) Learning Macromanagement in Starcraft by Deep Reinforcement Learning. Sensors 21(10):3332

    Article  Google Scholar 

  34. Irfan A, Zafar A, Hassan S (2019) Evolving levels for general games using deep convolutional generative adversarial networks. In 2019 11th Computer Science and Electronic Engineering (CEEC) pp 96–101

  35. Jeong Y, Choi H, Kim B, Gwon Y (2020) Defoggan: Predicting hidden information in the starcraft fog of war with generative adversarial nets. In Proceedings of the AAAI Conference on Artificial Intelligence 34(4):4296–4303

    Article  Google Scholar 

  36. Joo H-T, Lee S-H, Bae C-M, Kim K-J (2023) Learning to automatically spectate games for esports using object detection mechanism. Expert Syst Applic 213:118979

    Article  Google Scholar 

  37. Justesen N, Risi S (2017) Continual online evolutionary planning for in-game build order adaptation in StarCraft. In Proceedings of the Genetic and Evolutionary Computation Conference pp 187–194

  38. Justesen N, Risi S (2017) Learning macromanagement in starcraft from replays using deep learning. In 2017 IEEE Conference on Computational Intelligence and Games (CIG) pp 162–169

  39. Karavolos D, Liapis A,  Yannakakis GN (2018) Pairing character classes in a deathmatch shooter game via a deep-learning surrogate model. In Proceedings of the 13th international conference on the Foundations of digital games pp 1–10

  40. Khalifa A, Bontrager P, Earle S, Togelius J (2020) Pcgrl: Procedural content generation via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 16(1):95–101

  41. Khan MJ, Hassan S, Sukthankar G (2021) Leveraging Transformers for StarCraft Macromanagement Prediction. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) pp 1229–1234

  42. Köostler H, Gmeiner B (2013) A multi-objective genetic algorithm for build order optimization in starcraft ii. KI-Künstliche Intelligenz 27(3):221–233

    Article  Google Scholar 

  43. Lample G, Chaplot DS (2017) Playing FPS games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence 31(1)

  44. Lee CM, Ahn CW (2021) Feature extraction for starcraft ii league prediction. Electronics 10(8):909

    Article  Google Scholar 

  45. Lee D, Tang H, Zhang J, Xu H, Darrell T, Abbeel P (2018) Modular architecture for starcraft ii with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 14(1):187–193

  46. Lin Z, Gehring J, Khalidov V, Synnaeve G (2017) Stardata: A starcraft ai research dataset. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 13(1):50–56

    Google Scholar 

  47. Ling C, Tollmar K, Gisslén L (2020) Using deep convolutional neural networks to detect rendered glitches in video games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 16(1):66–73

  48. Liskowski P, Jáskowski W, Krawiec K (2018) Learning to play othello with deep neural networks. IEEE Trans Games 10(4):354–364

    Article  Google Scholar 

  49. Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In International conference on machine learning pp 1928–1937

  50. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Belle-mare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  51. Oh J, Chockalingam V, Lee H (2016) Control of memory, active perception, and action in minecraft. In International conference on machine learning, pp 2790-2799

  52. Olesen JK, Yannakakis GN, Hallam J (2008) Real-time challenge balance in an RTS game using rtNEAT. In 2008 IEEE Symposium On Computational Intelligence and Games pp 87–94

  53. Ontanón S, Synnaeve G, Uriarte A, Richoux F, Churchill D, Preuss M (2013) A survey of real-time strategy game ai research and competition in starcraft. IEEE Trans Comput Intell AI Games 5(4):293–311

    Article  Google Scholar 

  54. Ontañón S, Barriga NA, Silva CR, Moraes RO, Lelis LH (2018) The first microrts artificial intelligence competition. AI Mag 39(1):75–83

    Google Scholar 

  55. Rashid T, Samvelyan M, De Witt CS, Farquhar G, Foerster J, Whiteson S (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res 21(1):7234–7284

    MathSciNet  Google Scholar 

  56. Svendsen JB, Rathe EA (2012) Micromanagement in StarCraft using potential fields tuned with a multi-objective genetic algorithm. Norwegian University of Science and Technology

  57. Robertson G, Watson I (2014) A review of real-time strategy game ai. Ai Mag 35(4):75–104

    Google Scholar 

  58. Sánchez-Ruiz AA, Miranda M (2017) A machine learning approach to pre- dict the winner in starcraft based on influence maps. Entertain Comput 19:29–41

    Article  Google Scholar 

  59. Shen S, Fu Y, Su H, Pan H, Qiao P, Dou Y, Wang C (2021) Graphcomm: A graph neural network based method for multi-agent reinforcement learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 3510–3514

  60. Shin Y, Kim J, Jin K, Kim YB (2020) Playtesting in match 3 game using strategic plays via reinforcement learning. IEEE Access 8:51593–51600

    Article  Google Scholar 

  61. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driess-che G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  62. Soemers D (2014) Tactical planning using MCTS in the game of StarCraft. Doctoral dissertation, Department of Knowledge Engineering, Maastricht University

  63. Sopov V, Makarov I (2021) Transformer-Based Deep Reinforcement Learning in VizDoom. In International Conference on Analysis of Images, Social Networks and Texts, pp 96–110. Cham: Springer International Publishing

  64. Stanescu M, Barriga N, Buro M (2014) Hierarchical adversarial search applied to real-time strategy games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 10(1):66–72

    Google Scholar 

  65. Stanescu M, Barriga NA, Hess A, Buro M (2016) Evaluating real-time strategy game states using convolutional neural networks. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp 1–7

  66. Synnaeve G, Bessiere P (2012) Special tactics: A bayesian approach to tactical decision-making. In 2012 IEEE Conference on Computational Intelligence and Games (CIG) pp 409–416

  67. Synnaeve G, Bessiere P (2011) A Bayesian model for opening prediction in RTS games with application to StarCraft. In 2011 IEEE Conference on Computational Intelligence and Games (CIG'11), pp 281–288

  68. Takada K, Iizuka H, Yamamoto M (2017) Reinforcement learning for creating evaluation function using convolutional neural network in hex. In 2017 Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp 196–201

  69. Uriarte A, Ontanón S (2012) Kiting in RTS games using influence maps. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 8(3):31–36

    Google Scholar 

  70. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  71. Vazifedan A, Izadi M (2022) Predicting human behavior in size-variant repeated games through deep convolutional neural networks. Progress Artif Intell 11(1):15–28

    Article  Google Scholar 

  72. Vinyals O, Ewalds T, Bartunov S, Georgiev P, Vezhnevets AS, Yeo M, Tsing R (2017) Starcraft ii: A new challenge for reinforcement learning. https://arxiv.org/pdf/1708.04782.pdf

  73. Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P et al (2019) Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature 575(7782):350–354

    Article  Google Scholar 

  74. Watson N, Hendricks S, Stewart T, Durbach I (2021) Integrating machine learning and decision support in tactical decision-making in rugby union. J Oper Res Soc 72(10):2274–2285

    Article  Google Scholar 

  75. Xu S, Kuang H, Zhi Z, Hu R, Liu Y, Sun H (2019) Macro action selection with deep reinforcement learning in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 15(1):94–99

  76. Xu Y, Chen L, Fang M, Wang Y, Zhang C (2020) Deep reinforcement learning with transformers for text adventure games. In 2020 IEEE Conference on Games (CoG) pp 65–72

  77. Ye D, Chen G, Zhang W, Chen S, Yuan B, Liu B, Chen J, Liu Z, Qiu F, Yu H et al (2020) Towards playing full moba games with deep reinforcement learning. Adv Neural Inform Process Syst 33:621–632

    Google Scholar 

  78. Young J, Hawes N (2012) Evolutionary learning of goal priorities in a real-time strategy game. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 8(1):87–92

    Google Scholar 

  79. Young J, Smith F, Atkinson C, Poyner K, Chothia T (2012) Scail: An integrated starcraft ai system. In 2012 IEEE Conference on Computational Intelligence and Games (CIG), pp 438–445

  80. Zahavy T, Haroush M, Merlis N, Mankowitz DJ, Mannor S (2018) Learn what not to learn: Action elimination with deep reinforcement learning. Adv Neural Inf Process Syst 31

  81. Zhang S, Wu Y, Ogai H, Inujima H, Tateno S (2021) Tactical decisionmaking for autonomous driving using dueling double deep q network with double attention. IEEE Access 9:151983–151992

    Article  Google Scholar 

  82. Zhen JS, Watson I (2013) Neuroevolution for micromanagement in the real-time strategy game StarCraft: Brood War. In AI 2013: Advances in Artificial Intelligence: 26th Australasian Joint Conference, Dunedin, New Zealand, December 1-6, 2013. Proceedings 26 pp 259–270. Springer International Publishing

Download references

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government Ministry of Science and ICT (MSIT) (2021R1A4A1030075).

Author information

Authors and Affiliations

  1. School of Integrated Technology, Gwangju Institute of Science and Technology, 123, Cheomdangwagi-ro, Buk-gu, 61005, Gwangju, South Korea

    Isaac Han & Kyung-Joong Kim

Authors
  1. Isaac Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kyung-Joong Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kyung-Joong Kim.

Ethics declarations

Conflict of interests

Authors have no conflicts of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 284 KB)

Supplementary file2 (WMV 370784 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, I., Kim, KJ. Deep ensemble learning of tactics to control the main force in a real-time strategy game. Multimed Tools Appl 83, 12059–12087 (2024). https://doi.org/10.1007/s11042-023-15742-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15742-x

Keywords

Search

Navigation