Abstract
Recent research focuses on how agents can learn to communicate with each other. This communication between the agents allows them to share information and coordinate their behaviour. Recent efforts have proven successful in these cooperative problems. A major problem we face in multi-agent reinforcement learning is the lazy agent problem, where some agents take advantage of the successful actions of other agents. This results in agents not being able to learn a functional policy. In this paper we will combine state-of-the-art methods to design an architecture to address cooperative problems using communication while also eliminating the lazy agent problem. We propose two approaches for learning to communicate that use value decomposition to address the lazy agent problem. We find that the additive version of value decomposition gives us results which exceeds the results of the state of the art.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Foerster, J., Assael, I.A., de Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. In: Advances in Neural Information Processing Systems, pp. 2137–2145 (2016)
Sukhbaatar, S., Fergus, R., et al.: Learning multiagent communication with backpropagation. In: Advances in Neural Information Processing Systems, pp. 2244–2252 (2016)
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K., et al.: Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press (2018)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)
Bellman, R., et al.: The theory of dynamic programming. Bull. Am. Math. Soc. 60(6), 503–515 (1954)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Hausknecht, M., Stone, P.: Deep recurrent q-learning for partially observable MDPs. In: 2015 AAAI Fall Symposium Series (2015)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Neural Information Processing Systems (NIPS) (2017)
Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908 (2017)
Acknowledgement
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Vanneste, S., Vanneste, A., Bosmans, S., Mercelis, S., Hellinckx, P. (2020). Learning to Communicate with Multi-agent Reinforcement Learning Using Value-Decomposition Networks. In: Barolli, L., Hellinckx, P., Natwichai, J. (eds) Advances on P2P, Parallel, Grid, Cloud and Internet Computing. 3PGCIC 2019. Lecture Notes in Networks and Systems, vol 96. Springer, Cham. https://doi.org/10.1007/978-3-030-33509-0_69
Download citation
DOI: https://doi.org/10.1007/978-3-030-33509-0_69
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33508-3
Online ISBN: 978-3-030-33509-0
eBook Packages: EngineeringEngineering (R0)