Multi-agent reinforcement learning based on local communication

Article
  • 41 Downloads

Abstract

Aiming at the locality and uncertainty of observations in large-scale multi-agent application scenarios, the model of Decentralized Partially Observable Markov Decision Processes (DEC-POMDP) is considered, and a novel multi-agent reinforcement learning algorithm based on local communication is proposed. For a distributed learning environment, the elements of reinforcement learning are difficult to describe effectively in local observation situation, and the learning behaviour of each individual agent is influenced by its teammates. The local communication with consensus protocol is utilized to agree on the global observing environment, and thus that a part of strategies generated by repeating observations are eliminated, and the agent team gradually reach uniform opinion on the state of the event or object to be observed, they can thus approach a unique belief space regardless of whether each individual agent can perform a complete or partial observation. The simulation results show that the learning strategy space is reduced, and the learning speed is improved.

Keywords

Reinforcement learning Multi-agent Local communication Consensus 

References

  1. 1.
    Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34(6), 96–108 (2017)CrossRefGoogle Scholar
  2. 2.
    Girard, J., Emami, M.R.: Concurrent Markov decision processes for robot team learning. Eng. Appl. Artif. Intell. 39, 223–234 (2015)CrossRefGoogle Scholar
  3. 3.
    Shani, G., Pineau, J., Kaplow, R.: A survey of point-based POMDP solvers. Auton. Agent. Multi-Agent Syst. 27(1), 1–51 (2013)CrossRefGoogle Scholar
  4. 4.
    Pajarinen, J., Hottinen, A., Peltonen, J.: Optimizing spatial and temporal reuse in wireless networks by decentralized partially observable Markov decision processes. IEEE Trans. Mob. Comput. 13(4), 866–879 (2014)CrossRefGoogle Scholar
  5. 5.
    Kraemer, L., Banerjee, B.: Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing 190, 82–94 (2016)CrossRefGoogle Scholar
  6. 6.
    Kraemer, L., Banerjee, B.: Reinforcement learning of informed initial policies for decentralized planning. ACM Trans. Auton. Adapt. Syst. 9(4), 1–32 (2015)CrossRefGoogle Scholar
  7. 7.
    Sharma, R., Spaan, M.T.J.: Bayesian-game-based fuzzy reinforcement learning control for decentralized POMDPs. IEEE Trans. Comput. Intell. AI Games 4(4), 309–328 (2012)CrossRefGoogle Scholar
  8. 8.
    Vaisenberg, R., Della Motta, A., Mehrotra, S., et al.: Scheduling sensors for monitoring sentient spaces using an approximate POMDP policy. Pervasive Mobile Comput. 10, 83–103 (2014)CrossRefGoogle Scholar
  9. 9.
    Chandrasekaran, M., Doshi, P., Zeng, Y., et al.: Can bounded and self-interested agents be teammates? Application to planning in ad hoc teams. Auton. Agent. Multi-Agent Syst. 31(4), 821–860 (2017)CrossRefGoogle Scholar
  10. 10.
    Dutta, P.S., Jennings, N.R., Moreau, L.: Cooperative information sharing to improve distributed learning in multi-agent systems. J. Artif. Intell. Res. 24, 407–463 (2005)MATHGoogle Scholar
  11. 11.
    Fang, M., Groen, F.C.A., Li, H., et al.: Collaborative multi-agent reinforcement learning based on a novel coordination tree frame with dynamic partition. Eng. Appl. Artif. Intell. 27(1), 191–198 (2014)CrossRefGoogle Scholar
  12. 12.
    Cockburn, J., Collins, A.E., Frank, M.: A reinforcement learning mechanism responsible for the valuation of free choice. Neuron 83(3), 551–557 (2014)CrossRefGoogle Scholar
  13. 13.
    Mongillo, G., Shteingart, H., Loewenstein, Y.: The misbehavior of reinforcement learning. Proc. IEEE 102(4), 528–541 (2014)CrossRefGoogle Scholar
  14. 14.
    La, H.M., Sheng, W.: Distributed sensor fusion for scalar field mapping using mobile sensor networks. IEEE Trans. Cybern. 43(2), 766–778 (2013)CrossRefGoogle Scholar
  15. 15.
    Liu, J., Anderson, B.D.O., Cao, M., et al.: Analysis of accelerated gossip algorithms. Automatica 49(4), 873–883 (2013)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Wang, Y.H., Li, T.H.S., Lin, C.J.: Backward Q-learning: the combination of Sarsa algorithm and Q-learning. Eng. Appl. Artif. Intell. 26(9), 2184–2193 (2013)CrossRefGoogle Scholar
  17. 17.
    Mazzarella, L., Sarlette, A., Ticozzi, F.: Consensus for quantum networks: symmetry from gossip interactions. IEEE Trans. Autom. Control 60(1), 158–172 (2015)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Liu, D., Wei, Q.: Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learning Syst. 25(3), 621–634 (2014)CrossRefGoogle Scholar
  19. 19.
    Pynadath, D.V., Tambe, M.: The communicative multi-agent team decision problem: analyzing teamwork theories and models. J. Artif. Intell. Res. 16, 389–423 (2002)MATHGoogle Scholar
  20. 20.
    Cao, Y., Yu, W., Ren, W., et al.: An overview of recent progress in the study of distributed multi-agent coordination. IEEE Trans. Industr. Inf. 9(1), 427–438 (2013)CrossRefGoogle Scholar
  21. 21.
    Wu, F., Zilberstein, S., Chen, X.: Online planning for multi-agent systems with bounded communication. Artif. Intell. 175(2), 487–511 (2011)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Electrical EngineeringSouthwest Jiaotong UniversityChengduChina

Personalised recommendations