Abstract
Free-energy based reinforcement learning was proposed for learning in high-dimensional state and action spaces, which cannot be handled by standard function approximation methods in reinforcement learning. In the free-energy reinforcement learning method, the action-value function is approximated as the negative free energy of a restricted Boltzmann machine. In this paper, we test if it is feasible to use free-energy reinforcement learning for real robot control with raw, high-dimensional sensory inputs through the extraction of task-relevant features in the hidden layer. We first demonstrate, in simulation, that a small mobile robot could efficiently learn a vision-based navigation and battery capturing task. We then demonstrate, for a simpler battery capturing task, that free-energy reinforcement learning can be used for on-line learning in a real robot. The analysis of learned weights showed that action-oriented state coding was achieved in the hidden layer.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Sutton, R.S., Barto, A.: Reinforcement Learning: An Introduction (1998)
Sallans, B., Hinton, G.E.: Reinforcement learning with factored states and actions. Journal of Machine Learning Research 5, 1063–1088 (2004)
Otsuka, M., Yoshimoto, J., Doya, K.: Free-energy-based reinforcement learning in apartially observable environments. In: Proceedings of ESANN, pp. 541–545 (2010)
Shibata, K., Iida, M.: Acquisition of box pushing by direct-vision-based reinforce-ment learning. In: Proceedings of SICE 2003, vol. 3, pp. 2322–2327 (2003)
Shibata, K., Kawano, T.: Learning of action generation from raw camera imagesin a real-world-like environment by simple coupling of reinforcement learning and a neural network. In: Köppen, M., Kasabov, N., Coghill, G. (eds.) ICONIP 2008. LNCS, vol. 5506, pp. 755–762. Springer, Heidelberg (2009)
Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, Cambridge University (1994)
Sutton, R.S.: Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: NIPS 1995, pp. 1038–1044 (1996)
Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Machine Learning 22(1-3), 123–158 (1996)
Doya, K., Uchibe, E.: The cyber rodent project: Exploration of adaptive mechanisms for self-preservation and self-reproduction. Adaptive Behavior 13(2), 149–160 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Elfwing, S., Otsuka, M., Uchibe, E., Doya, K. (2010). Free-Energy Based Reinforcement Learning for Vision-Based Navigation with High-Dimensional Sensory Inputs. In: Wong, K.W., Mendis, B.S.U., Bouzerdoum, A. (eds) Neural Information Processing. Theory and Algorithms. ICONIP 2010. Lecture Notes in Computer Science, vol 6443. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17537-4_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-17537-4_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17536-7
Online ISBN: 978-3-642-17537-4
eBook Packages: Computer ScienceComputer Science (R0)