Towards a Deep Reinforcement Learning Approach for Tower Line Wars

  • Per-Arne Andersen
  • Morten Goodwin
  • Ole-Christoffer Granmo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10630)


There have been numerous breakthroughs with reinforcement learning in the recent years, perhaps most notably on Deep Reinforcement Learning successfully playing and winning relatively advanced computer games. There is undoubtedly an anticipation that Deep Reinforcement Learning will play a major role when the first AI masters the complicated game plays needed to beat a professional Real-Time Strategy game player. For this to be possible, there needs to be a game environment that targets and fosters AI research, and specifically Deep Reinforcement Learning. Some game environments already exist, however, these are either overly simplistic such as Atari 2600 or complex such as Starcraft II from Blizzard Entertainment.

We propose a game environment in between Atari 2600 and Starcraft II, particularly targeting Deep Reinforcement Learning algorithm research. The environment is a variant of Tower Line Wars from Warcraft III, Blizzard Entertainment. Further, as a proof of concept that the environment can harbor Deep Reinforcement algorithms, we propose and apply a Deep Q-Reinforcement architecture. The architecture simplifies the state space so that it is applicable to Q-learning, and in turn improves performance compared to current state-of-the-art methods. Our experiments show that the proposed architecture can learn to play the environment well, and score 33% better than standard Deep Q-learning—which in turn proves the usefulness of the game environment.


Reinforcement Learning Q-Learning Deep Learning Game environment 


  1. 1.
    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing ATARI with deep reinforcement learning. In: NIPS Deep Learning Workshop (2013)Google Scholar
  2. 2.
    Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A.J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., Kumaran, D., Hadsell, R.: Learning to navigate in complex environments. CoRR abs/1611.03673 (2016)Google Scholar
  3. 3.
    van Seijen, H., Fatemi, M., Romoff, J., Laroche, R., Barnes, T., Tsang, J.: Hybrid reward architecture for reinforcement learning. abs/1706.04208 (2017)Google Scholar
  4. 4.
    Gosavi, A.: Reinforcement learning: a tutorial survey and recent advances. INFORMS J. Comput. 21(2), 178–192 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. CoRR abs/1509.06461 (2015)Google Scholar
  6. 6.
    Wang, Z., de Freitas, N., Lanctot, M.: Dueling network architectures for deep reinforcement learning. CoRR abs/1511.06581 (2015)Google Scholar
  7. 7.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)Google Scholar
  8. 8.
    Traysent: Starcraft ii api - technical design, November.
  9. 9.
    Vinyals, O.: Deepmind and blizzard to release starcraft ii as an ai research environment, November 2016.
  10. 10.
    Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. CoRR abs/1509.02971 (2015)Google Scholar
  11. 11.
    Uriarte, A., Ontañón, S.: Game-tree search over high-level game states in RTS games, October 2014Google Scholar
  12. 12.
    Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. CoRR abs/1207.4708 (2012)Google Scholar
  13. 13.
    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. CoRR abs/1511.05952 (2015)Google Scholar
  14. 14.
    Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Sasha Vezhnevets, A., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., Quan, J., Gaffney, S., Petersen, S., Simonyan, K., Schaul, T., van Hasselt, H., Silver, D., Lillicrap, T., Calderone, K., Keet, P., Brunasso, A., Lawrence, D., Ekermo, A., Repp, J., Tsing, R.: StarCraft II: a new challenge for reinforcement learning. ArXiv e-prints, August 2017Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Per-Arne Andersen
    • 1
  • Morten Goodwin
    • 1
  • Ole-Christoffer Granmo
    • 1
  1. 1.University of AgderGrimstadNorway

Personalised recommendations