Abstract
Deep reinforcement learning (DRL) is one of the promising approaches to make robots accomplish complicated tasks. Eligibility traces is well known as a online learning technique to improve sample efficiency in the traditional reinforcement learning with linear regressors, not DRL. This is because dependencies between parameters of deep neural networks would destroy the eligibility traces. To resolve this problem, this study proposes a new eligibility traces method that makes it possible to be applied even into DRL. The eligibility traces in DRL accumulate gradients computed based on the past parameters, which are different from that computed based on the latest parameters. Hence, the proposed method considers the divergence between the past and latest parameters to adaptively decay the eligibility traces. Instead of that divergence directly, Bregman divergences between outputs computed by the past and latest parameters, which are computationally feasible, are exploited. In benchmark tasks for DRL, the proposed method acquires the tasks stably in comparison to the cases without/with the standard eligibility traces.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
Brockman, G., et al.: Openai gym. arXiv preprint arXiv:1606.01540 (2016)
Coumans, E., Bai, Y.: Pybullet, a python module for physics simulation for games, robotics and machine learning. GitHub repository (2016)
Elfwing, S., Uchibe, E., Doya, K.: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Netw. 107, 3–11 (2018)
Hayes, T.L., Cahill, N.D., Kanan, C.: Memory efficient experience replay for streaming learning. In: International Conference on Robotics and Automation, pp. 9769–9776. IEEE (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kobayashi, T.: Student-t policy in reinforcement learning to acquire global optimum of robot control. Appl. Intell. 49(12), 4335–4347 (2019). https://doi.org/10.1007/s10489-019-01510-8
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., Quillen, D.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robot. Res. 37(4–5), 421–436 (2018)
Lin, L.J.: Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn. 8(3–4), 293–321 (1992)
Modares, H., Ranatunga, I., Lewis, F.L., Popa, D.O.: Optimized assistive human-robot interaction using reinforcement learning. IEEE Trans. Cybern. 46(3), 655–667 (2015)
Parisi, S., Tangkaratt, V., Peters, J., Khan, M.E.: Td-regularized actor-critic methods. Machine Learning, pp. 1–35 (2019)
Paszke, A., et al.: Automatic differentiation in pytorch. In: Advances in Neural Information Processing Systems Workshop (2017)
Ramachandran, P., Zoph, B., Le, Q.V.: Swish: a self-gated activation function. arXiv preprint arXiv:1710.05941 7 (2017)
Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation. In: International Conference on Learning Representations (2016)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
van Seijen, H.: Effective multi-step temporal-difference learning for non-linear function approximation. arXiv preprint arXiv:1608.05151 (2016)
Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484 (2016)
Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Mach. Learn. 22(1–3), 123–158 (1996)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (2018)
Tokdar, S.T., Kass, R.E.: Importance sampling: a review. Wiley Interdisciplinary Rev. Comput. Stat. 2(1), 54–60 (2010)
Tsurumine, Y., Cui, Y., Uchibe, E., Matsubara, T.: Deep reinforcement learning with smooth policy update: application to robotic cloth manipulation. Robot. Auton. Syst. 112, 72–83 (2019)
Van Seijen, H., Mahmood, A.R., Pilarski, P.M., Machado, M.C., Sutton, R.S.: True online temporal-difference learning. J. Mach. Learn. Res. 17(1), 5057–5096 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kobayashi, T. (2022). Adaptive Eligibility Traces for Online Deep Reinforcement Learning. In: Ang Jr, M.H., Asama, H., Lin, W., Foong, S. (eds) Intelligent Autonomous Systems 16. IAS 2021. Lecture Notes in Networks and Systems, vol 412. Springer, Cham. https://doi.org/10.1007/978-3-030-95892-3_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-95892-3_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95891-6
Online ISBN: 978-3-030-95892-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)