Abstract
This research project presents a multi agent architecture which uses reinforcement learning. The goal is to design a system that able the agents to take advantage of its peers’ knowledge. The knowledge of the environment is obtained from the reinforcement learning algorithm, Q-learning. While, the multi agent architecture sets a communication model among the agents of the system. To reach the goal, the present research project takes advantage of the Q-learning characteristic, off-policy, incorporating a condition before the use of ε-greedy. This condition allows the agents not to explore a state that has already been sent by another agent, or itself. In the proposed multi agent architecture the agents work in pairs. Each pair of agents have two different behaviors allowing them to communicate and work on relevant states of the environment. The conditions to send the states depend on the environment, specifically, it depends on the circumstances which the agent obtains a reward from the environment. The results evidences that the number of agent-environment interactions to improve agent’s behavior is reduced by more than 90% through the proposed architecture.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wooldridge, M.: An Introduction to Multi Agent Systems. Wiley, Chichester (2009)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. The MIT Press, Londres (2018)
Fusté, A., Amores, J., Ha, D., Jongejan, J., Pitaru, A.: Paper cubes: evolving 3D characters in augmented reality using recurrent neural networks. In: Workshop in Machine Learning for Creativity and Design, NIPS 2017 (2017). http://annafuste.com/portfolio/paper-cubes/
Kraemer, L., Banerjee, B.: Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing 190, 82–94 (2016). https://www.sciencedirect.com/science/article/abs/pii/S0925231216000783?via%3Dihub
Hwang, K.S., Lin, J.L., Hsu, H.P.: A multi-agent cooperation system based on a layered cooperation model. In: 2014 IEEE International Conference on System Science and Engineering (ICSSE), pp. 149–153 (2014). https://ieeexplore.ieee.org/document/6887923
Dolan, P., Hallsworth, M., Halpern, D., King, D., Vlaev, I.: MINDSPACE: influencing behaviour through public policy (2010). https://www.instituteforgovernment.org.uk/sites/default/files/publications/MINDSPACE.pdf
Kim, D., et al.: Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019). https://arxiv.org/abs/1902.01554
Foerster, J.N., Yannis, M.A., de Freitas, N., Whiteson, S.: Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016). https://papers.nips.cc/paper/6042-learning-to-communicate-with-deep-multi-agent-reinforcement-learning.pdf
Bonghez, S.: Wise people learn by other people’s mistakes, fools by their own… In: PMI® Global Congress (2009). https://www.pmi.org/learning/library/wise-people-learn-mistakes-knowledge-management-6862
Silva, F., Costa, A.: A survey on transfer learning for multiagent reinforcement learning systems. J. Artif. Intell. Res. 64, 645–703 (2019). https://jair.org/index.php/jair/article/view/11396
Griffith, S., Subramanian, K., Scholz, J., Isbell, C.L., Thomaz, A.L.: Policy shaping: integrating human feedback with reinforcement learning. In: Advances in Neural Information Processing Systems (NIPS), pp. 2625–2633. https://smartech.gatech.edu/handle/1853/53270
Torrey, L., Taylor, M.E.: Teaching on a budget: agents advising agents in reinforcement learning. In: Proceedings of 12th the International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 1053–1060 (2013). https://papers.nips.cc/paper/5187-policy-shaping-integrating-human-feedback-with-reinforcement-learning
Zhan, Y., Bou-Ammar, H., Taylor, M.E.: Theoretically-grounded policy advice from multiple teachers in reinforcement learning settings with applications to negative transfer. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI), pp. 2315–2321 (2016). doi:https://dl.acm.org/doi/10.5555/3060832.3060945
Amir, O., Kamar, E., Kolobov, A., Grosz, B.: Interactive teaching strategies for agent training. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI), pp. 804–811 (2016). https://dl.acm.org/doi/10.5555/3060621.3060733
Silva, F., Glatt, R., Costa, A.: Simultaneously learning and advising in multiagent reinforcement learning. In: Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1100–1108 (2017). https://dl.acm.org/doi/10.5555/3091125.3091280
Fachantidis, A., Taylor, M., Vlahavas, I.: Learning to teach reinforcement learning agents. In: Machine Learning and Knowledge Extraction, pp. 21–42 (2019). https://www.mdpi.com/2504-4990/1/1/2
Omidshafiei, S., et al.: Learning to teach in cooperative multiagent reinforcement learning. In: Workshop on Lifelong Learning: A Reinforcement Learning Approach (2018). https://arxiv.org/abs/1805.07830
Majumdar, A., Benavidez, P., Jamshidi, M.: Multi-agent exploration for faster and reliable deep q-learning convergence in reinforcement learning. In: 2018 World Automation Congress (WAC), pp. 1–6 (2018). https://ieeexplore.ieee.org/document/8430409
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Cárdenas Guilcapi, D.A., Paz-Arias, H., Galindo, J. (2020). Design and Implementation of a Multi Agent Architecture to Communicate Reinforcement Learning Knowledge and Improve Agents’ Behavior. In: Rodriguez Morales, G., Fonseca C., E.R., Salgado, J.P., Pérez-Gosende, P., Orellana Cordero, M., Berrezueta, S. (eds) Information and Communication Technologies. TICEC 2020. Communications in Computer and Information Science, vol 1307. Springer, Cham. https://doi.org/10.1007/978-3-030-62833-8_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-62833-8_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62832-1
Online ISBN: 978-3-030-62833-8
eBook Packages: Computer ScienceComputer Science (R0)