Abstract
This paper presents a navigation component based on a hybrid case-based reasoning (CBR) and reinforcement learning (RL) approach for an AI agent in a real-time strategy (RTS) game. Spatial environment information is abstracted into a number of influence maps. These influence maps are then combined into cases that are managed by the CBR component. RL is used to update the case solutions which are composed of unit actions with associated fitness values. We present a detailed account of the architecture and underlying model. Our model accounts for all relevant environment influences with a focus on two main subgoals: damage avoidance and target approximation. For each of these subgoals, we create scenarios in the StarCraft RTS game and look at the performance of our approach given different similarity thresholds for the CBR part. The results show, that our navigation component manages to learn how to fulfill both sub-goals given the choice of a suitable similarity threshold. Finally, we combine both subgoals for the overall navigation component and show a comparison between the integrated approach, a random action selection, and a target-selection-only agent. The results show that the CBR/RL approach manages to successfully learn how to navigate towards goal positions while at the same time avoiding enemy attacks.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Aamodt, A., Plaza, E.: Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Communications 7(1), 39–59 (1994)
Bellman, R., Bellman, R.E., Bellman, R.E., Bellman, R.E.: Adaptive control processes: a guided tour, vol. 4. Princeton University Press, Princeton (1961)
Brafman, R.I., Tennenholtz, M.: R-max-a general polynomial time algorithm for near-optimal reinforcement learning. The Journal of Machine Learning Research 3, 213–231 (2003)
Bulitko, V., Bjornsson, Y., Lawrence, R.: Case-based subgoaling in real-time heuristic search for video game pathfinding. Journal of Artificial Intelligence Research 39, 269–300 (2010)
Buro, M., Furtak, T.: Rts games and real-time ai research. In: Proceedings of the Behavior Representation in Modeling and Simulation Conference (BRIMS), pp. 63–70. Citeseer (2004)
Hagelbäck, J.: Multi-Agent Potential Field Based Architectures for Real-Time Strategy Game Bots. Ph.D. thesis, Blekinge Institute of Technology (2012)
Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics 4(2), 100–107 (1968)
Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. The International Journal of Robotics Research 5(1), 90–98 (1986)
Kruusmaa, M.: Global navigation in dynamic environments using case-based reasoning. Autonomous Robots 14(1), 71–91 (2003)
Laue, T., Spiess, K., Röfer, T.: SimRobot – A general physical robot simulator and its application in roboCup. In: Bredenfeld, A., Jacoff, A., Noda, I., Takahashi, Y. (eds.) RoboCup 2005. LNCS (LNAI), vol. 4020, pp. 173–183. Springer, Heidelberg (2006)
Molineaux, M., Aha, D., Moore, P.: Learning continuous action models in a real-time strategy environment. In: Proceedings of the Twenty-First Annual Conference of the Florida Artificial Intelligence Research Society, pp. 257–262 (2008)
Ontanón, S., Synnaeve, G., Uriarte, A., Richoux, F., Churchill, D., Preuss, M.: A survey of real-time strategy game ai research and competition in starcraft. IEEE Transactions on Computational Intelligence and AI in Games (2013)
Ram, A., Santamaria, J.C.: Continuous case-based reasoning. Artificial Intelligence 90(1), 25–77 (1997)
Smyth, B., Cunningham, P.: Déjà vu: A hierarchical case-based reasoning system for software design. In: ECAI, vol. 92, pp. 587–589 (1992)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)
Uriarte, A., Ontañón, S.: Kiting in rts games using influence maps. In: Workshop Proceedings of the Eighth Artificial Intelligence and Interactive Digital Entertainment Conference (2012)
Watkins, C.: Learning from Delayed Rewards. Ph.D. thesis. University of Cambridge, England (1989)
Weber, B., Mateas, M., Jhala, A.: Building human-level ai for real-time strategy games. In: 2011 AAAI Fall Symposium Series (2011)
Wender, S., Watson, I.: Applying reinforcement learning to small scale combat in the real-time strategy game starcraft: broodwar. In: 2012 IEEE Conference on Computational Intelligence and Games, CIG (2012)
Wess, S., Althoff, K., Derwand, G.: Using k-d trees to improve the retrieval step in case-based reasoning. In: Wess, S., Richter, M., Althoff, K.-D. (eds.) EWCBR 1993. LNCS, vol. 837, pp. 167–181. Springer, Heidelberg (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Wender, S., Watson, I. (2014). Combining Case-Based Reasoning and Reinforcement Learning for Unit Navigation in Real-Time Strategy Game AI. In: Lamontagne, L., Plaza, E. (eds) Case-Based Reasoning Research and Development. ICCBR 2014. Lecture Notes in Computer Science(), vol 8765. Springer, Cham. https://doi.org/10.1007/978-3-319-11209-1_36
Download citation
DOI: https://doi.org/10.1007/978-3-319-11209-1_36
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11208-4
Online ISBN: 978-3-319-11209-1
eBook Packages: Computer ScienceComputer Science (R0)