Abstract
In this article, we present an on-line variational Bayes (VB) method for the identification of linear state space models. The learning algorithm is implemented as alternate maximization of an on-line free energy, which can be used for determining the dimension of the internal state. We also propose a reinforcement learning (RL) method using this system identification method. Our RL method is applied to a simple automatic control problem. The result shows that our method is able to determine correctly the dimension of the internal state and to acquire a good control, even in a partially observable environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Attias, H.: A variational Bayesian framework for graphical models, Advances in Neural Information Processing Systems 12, pp. 206–212 (2000).
Dempster, A. P. et al.: Maximum likelihood from incomplete data via the EM algorithm, Journal of Royal Statistical Society B, Vol. 39, pp. 1–38 (1977).
Früwirth-Schnatter, S.: Bayesian model descrimination and Bayes factors for linear Gaussian state space models, Journal of Royal Statistical Society B, Vol. 57, pp. 237–246 (1995).
Ghahramani, Z. and Beal, M. J.: Propagation Algorithms for Variational Bayesian Learning, Advances in Neural Information Processing Systems 13 (2001).
Konda, V. R.: Actor-Critic Algorithms, PhD Thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology (2002).
Roweis, S. and Ghahramani, Z.: A Unifying Review of Linear Gaussian Models, Neural Computation, Vol. 11, pp. 305–345 (1999).
Sato, M.: Online model selection based on the variational Bayes, Neural Computation, Vol. 13, No. 7, pp. 1649–1681 (2001).
Singh, S. P. et al.: Learning without state-estimation in partially observable Markovian decision processes, Proceedings of the 11th International Conference on Machine Learning, pp. 284–292 (1994).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yoshimoto, J., Ishii, S., Sato, Ma. (2003). System Identification Based on Online Variational Bayes Method and Its Application to Reinforcement Learning. In: Kaynak, O., Alpaydin, E., Oja, E., Xu, L. (eds) Artificial Neural Networks and Neural Information Processing — ICANN/ICONIP 2003. ICANN ICONIP 2003 2003. Lecture Notes in Computer Science, vol 2714. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44989-2_16
Download citation
DOI: https://doi.org/10.1007/3-540-44989-2_16
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40408-8
Online ISBN: 978-3-540-44989-8
eBook Packages: Springer Book Archive