Decentralized Reinforcement Learning Applied to Mobile Robots
In this paper, decentralized reinforcement learning is applied to a control problem with a multidimensional action space. We propose a decentralized reinforcement learning architecture for a mobile robot, where the individual components of the commanded velocity vector are learned in parallel by separate agents. We empirically demonstrate that the decentralized architecture outperforms its centralized counterpart in terms of the learning time, while using less computational resources. The method is validated on two problems: an extended version of the 3-dimensional mountain car, and a ball-pushing behavior performed with a differential-drive robot, which is also tested on a physical setup.
KeywordsMultiagent learning Decentralized control Reinforcement learning Robot soccer
This work was partially funded by FONDECYT under Project Number 1161500. David Leonardo Leottau was funded under grant CONICYT-PCHA/Doctorado Nacional/2013-63130183. The authors would like to thank Technical University of Delft for providing the resources to test the learnt policies on an experimental setup.
- 1.Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
- 3.Martin, J., Lope, H.D.: A distributed reinforcement learning architecture for multi-link robots. In: 4th International Conference on Informatics in Control, Automation and Robotics, ICINCO 2007. Number 3, Angers, Francia, pp. 192–197 (2007)Google Scholar
- 5.Busoniu, L., Schutter, B.D., Babuska, R.: Decentralized reinforcement learning control of a robotic manipulator. In: Ninth International Conference on Control, Automation, Robotics and Vision, ICARCV 2006, 5–8 December, Singapore, pp. 1–6. IEEE (2006)Google Scholar
- 7.Taylor, M.E., Kuhlmann, G., Stone, P.: Autonomous transfer for reinforcement learning. In: The Autonomous Agents and Multi-agent Systems Conference (AAMAS), Number May, Estoril, Portugal, pp. 283–290 (2008)Google Scholar
- 8.Systems, M.: Miabotpro manual (2016)Google Scholar
- 9.Leottau, D.L., Ruiz-del-Solar, J., MacAlpine, P., Stone, P.: A study of layered learning strategies applied to individual behaviors in robot soccer. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.) RoboCup 2015. LNCS (LNAI), vol. 9513, pp. 290–302. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-29339-4_24CrossRefGoogle Scholar
- 10.Takahashi, Y., Asada, M.: Multi-layered learning system for real robot behavior acquisition. In: Kordic, V., (ed.) Cutting Edge Robotics, Number pp. 357–375, July 2005 (2004)Google Scholar
- 11.Emery, R., Balch, T.: Behavior-based control of a non-holonomic robot in pushing tasks. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), vol. 3, pp. 2381–2388. IEEE (2001)Google Scholar
- 13.Vatsyayan, A.: Video: centralized and decentralized reinforcement learning of the ball-pushing behavior (2016). https://youtu.be/pajMkrf7ldY
- 14.Dziomin, U., Kabysh, A., Golovko, V., Stetter, R.: A multi-agent reinforcement learning approach for the efficient control of mobile robot. In: 2013 IEEE 7th International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS), vol. 2, pp. 867–873. IEEE (2013)Google Scholar
- 15.Leottau, D.L., Ruiz-del-Solar, J.: An accelerated approach to decentralized reinforcement learning of the Ball-Dribbling behavior. In: AAAI Workshops, Austin, Texas USA, pp. 23–29 (2015)Google Scholar
- 16.Kabysh, A., Golovko, V., Lipnickas, A.: Influence learning for multi-agent system based on reinforcement learning. Int. J. Comput. 11(1), 39–44 (2012)Google Scholar