Skip to main content

Learning to Communicate with Multi-agent Reinforcement Learning Using Value-Decomposition Networks

Part of the Lecture Notes in Networks and Systems book series (LNNS,volume 96)


Recent research focuses on how agents can learn to communicate with each other. This communication between the agents allows them to share information and coordinate their behaviour. Recent efforts have proven successful in these cooperative problems. A major problem we face in multi-agent reinforcement learning is the lazy agent problem, where some agents take advantage of the successful actions of other agents. This results in agents not being able to learn a functional policy. In this paper we will combine state-of-the-art methods to design an architecture to address cooperative problems using communication while also eliminating the lazy agent problem. We propose two approaches for learning to communicate that use value decomposition to address the lazy agent problem. We find that the additive version of value decomposition gives us results which exceeds the results of the state of the art.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. Foerster, J., Assael, I.A., de Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. In: Advances in Neural Information Processing Systems, pp. 2137–2145 (2016)

    Google Scholar 

  2. Sukhbaatar, S., Fergus, R., et al.: Learning multiagent communication with backpropagation. In: Advances in Neural Information Processing Systems, pp. 2244–2252 (2016)

    Google Scholar 

  3. Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W.M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J.Z., Tuyls, K., et al.: Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017)

  4. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press (2018)

    Google Scholar 

  5. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)

    MATH  Google Scholar 

  6. Bellman, R., et al.: The theory of dynamic programming. Bull. Am. Math. Soc. 60(6), 503–515 (1954)

    CrossRef  MathSciNet  Google Scholar 

  7. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)

    CrossRef  Google Scholar 

  8. Hausknecht, M., Stone, P.: Deep recurrent q-learning for partially observable MDPs. In: 2015 AAAI Fall Symposium Series (2015)

    Google Scholar 


  10. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Neural Information Processing Systems (NIPS) (2017)

    Google Scholar 

  11. Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908 (2017)

Download references


We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Simon Vanneste .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vanneste, S., Vanneste, A., Bosmans, S., Mercelis, S., Hellinckx, P. (2020). Learning to Communicate with Multi-agent Reinforcement Learning Using Value-Decomposition Networks. In: Barolli, L., Hellinckx, P., Natwichai, J. (eds) Advances on P2P, Parallel, Grid, Cloud and Internet Computing. 3PGCIC 2019. Lecture Notes in Networks and Systems, vol 96. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33508-3

  • Online ISBN: 978-3-030-33509-0

  • eBook Packages: EngineeringEngineering (R0)