Abstract
Multi agent strategies in mixed cooperative-competitive environments can be hard to craft by hand because each agent needs to coordinate with its teammates while competing with its opponents. Learning based algorithms are appealing but they require a competitive opponent to train against, which is often not available. Many scenarios require heterogeneous agent behavior for the team’s success and this increases the complexity of the learning algorithm. In this work, we develop a mixed cooperative-competitive multi agent environment called FortAttack in which two teams compete against each other for success. We show that modeling agents with Graph Neural Networks (GNNs) and training them with Reinforcement Learning (RL) from scratch, leads to the co-evolution of increasingly complex strategies for each team. Through competition in Multi-Agent Reinforcement Learning (MARL), we observe a natural emergence of heterogeneous behavior among homogeneous agents when such behavior can lead to the team’s success. Such heterogeneous behavior from homogeneous agents is appealing because any agent can replace the role of another agent at test time. Finally, we propose ensemble training, in which we utilize the evolved opponent strategies to train a single policy for friendly agents. We were able to train a large number of agents on a commodity laptop, which shows the scalability and efficiency of our approach. The code and a video presentation are available online (Code: https://github.com/Ankur-Deka/Emergent-Multiagent-Strategies, Video: https://youtu.be/ltHgKYc0F-E).
Supported by organization ONR N00014-19-C-1070, AFOSR/AFRL award FA9550-18-1-0251, Darpa DARPA Cooperative Agreement No.: HR00111820051 and AFOSR FA9550-15-1-0442.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agarwal, A., Kumar, S., Sycara, K.: Learning transferable cooperative behavior in multi-agent teams. arXiv preprint arXiv:1906.01202 (2019)
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., Abbeel, P.: Continuous adaptation via meta-learning in nonstationary and competitive environments. arXiv preprint arXiv:1710.03641 (2017)
Baker, B., et al.: Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528 (2019)
Brockman, G., et al.: Openai gym. arXiv preprint arXiv:1606.01540 (2016)
Bu, L., Babu, R., De Schutter, B., et al.: A comprehensive survey of multiagent reinforcement learning. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 38(2), 156–172 (2008)
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 (2018)
Han, L., et al.: Grid-wise control for multi-agent reinforcement learning in video game ai. In: International Conference on Machine Learning, pp. 2576–2585 (2019)
Hoshen, Y.: Vain: attentional multi-agent predictive modeling. In: Advances in Neural Information Processing Systems, pp. 2701–2711 (2017)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Neural Information Processing Systems (NIPS) (2017)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Ramachandran, R.K., Preiss, J.A., Sukhatme, G.S.: Resilience by reconfiguration: Exploiting heterogeneity in robot teams. arXiv preprint arXiv:1903.04856 (2019)
Rashid, T., Samvelyan, M., De Witt, C.S., Farquhar, G., Foerster, J., Whiteson, S.: Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485 (2018)
Samvelyan, M., et al .: The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043 (2019)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Shishika, D., Paulos, J., Kumar, V.: Cooperative team strategies for multi-player perimeter-defense games. IEEE Rob. Autom. Lett. 5(2), 2738–2745 (2020)
Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)
Sukhbaatar, S., Fergus, R., et al.: Learning multiagent communication with backpropagation. In: Advances in Neural Information Processing Systems, pp. 2244–2252 (2016)
Tampuu, A., et al.: Multiagent cooperation and competition with deep reinforcement learning. PLoS one 12(4), e0172395 (2017)
Tan, M.: Multi-agent reinforcement learning: independent vs. cooperative agents. In: Proceedings of the tenth International Conference on Machine Learning, pp. 330–337 (1993)
Vinyals, O., et al.: Starcraft ii: a new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017)
Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)
Wen, Y., Yang, Y., Luo, R., Wang, J., Pan, W.: Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv:1901.09207 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Deka, A., Sycara, K. (2021). Natural Emergence of Heterogeneous Strategies in Artificially Intelligent Competitive Teams. In: Tan, Y., Shi, Y. (eds) Advances in Swarm Intelligence. ICSI 2021. Lecture Notes in Computer Science(), vol 12689. Springer, Cham. https://doi.org/10.1007/978-3-030-78743-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-78743-1_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78742-4
Online ISBN: 978-3-030-78743-1
eBook Packages: Computer ScienceComputer Science (R0)