Abstract
In this paper, we present nonlinear Bayesian filters for training recurrent neural networks with a special emphasis on a novel, more accurate, derivative-free member of the approximate Bayesian filter family called the cubature Kalman filter. We discuss the theory of Bayesian filters, which is rooted in the state-space modeling of the dynamic system in question and the linear estimation principle. For improved numerical stability and optimal performance during training period, a number of techniques of how to tune Bayesian filters is suggested. We compare the predictability of various Bayesian filter-trained recurrent neural networks using a chaotic time-series. From the empirical results, we conclude that the performance may be greatly improved by the new square-root cubature Kalman filter.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Anderson, B.D.O., Moore, J.B.: Optimal Filtering. Prentice Hall, New Jersey (1979)
Arasaratnam, I., Haykin, S.: Cubature Kalman Filters. IEEE Trans. Automatic Cont. (2008) (under 3rd review)
Bengio, Y., Simard, P., Frasconi, P.: Learning Long-term Dependencies with Gradient Descent is Difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)
Cappe, O., Godsil, S.J., Moulines, E.: An Overview of Existing Methods and Recent Advances in Sequential Monte Carlo. Proc. IEEE 95(5) (2007)
Gordon, N.J., Salmond, D.J., Smith, A.F.M.: Novel Approach to Nonlinear/Non-Gaussian Bayesian State Estimation. IEE Proc. -F. 140, 107–113 (1993)
Hanson, S.J., Burr, D.J.: Minkowski-r Back-Propagation: Learning in Connectionist Models with Non-Euclidian Error Signals. In: Anderson, D. (ed.) Neural inform. proces. sys., pp. 348–357. American Inst. of Physics, NY (1988)
Haykin, S., Li, L.: Nonlinear Adaptive Prediction of Nonstationary Signals. IEEE Trans. Signal Process 43(2), 526–535 (1995)
Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall, New Jersey (1999)
Haykin, S. (ed.): Kalman Filtering and Neural Networks. Wiley, New York (2001)
Julier, S.J., Ulhmann, J.K., Durrant-Whyte, H.F.: A New Method for Nonlinear Transformation of Means and Covariances in Filters and Estimators. IEEE Trans. Automatic Cont. 45, 472–482 (2000)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-Based Learning Applied to Document Recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Liu, J.S.: Monte Carlo Strategies in Scientific Computing. Springer, Heidelberg (2001)
Narendra, K.S., Parthasarathy, K.: Identification and Control of Dynamical Systems Using Neural Networks. IEEE Trans. Neural Netw. 1(1), 4–27 (1990)
Nørgaard, M.N., Poulsen, N.K., Ravn, O.: New Developments in State Estimation of Nonlinear Systems. Automatica 36, 1627–1638 (2000)
Ott, E.: Chaos in Dynamical Systems, 2nd edn. Cambridge Univ. Press, Cambridge (2002)
Peterka, V.: Bayesian Approach to System Identification. In: Eykhoff, P. (ed.) Trends and progress in system identification, pp. 239–304. Pergamon Press, Oxford (1981)
Pineda, F.: Generalization of Backpropagation to Recurrent Neural Networks. Physical Review Letters 59(19) (1987)
Plumer, E.S.: Training Neural Networks Using Sequential Extened Kalman Filters. In: Proc. World Congress on Neural Netw., Washigton, DC, pp. I-764–I-769 (1995)
Prokhorov, D.: Training Recurrent Neurocontrollers for Robustness with Derivative-Free Kalman Filter. IEEE Trans. Neural Netw. 17(6), 1606–1616 (2006)
Puskorious, G., Feldkamp, L.: Neurocontrol of Nonlinear Dynamical Systems with Kalman Filter Trained Neural Networks. IEEE Trans. Neural Netw. 5(2) (1994)
Ruck, D.W., Rogers, S.K., Kabrisky, M., Maybeck, P., Oxle, M.E.: Comparative Analysis of Backpropgation and the Extended Kalman Filter for Training Multilayer Perceptrons. IEEE Trans. Patt. Anal. & Mach. Intell. 14(6), 686–691 (1992)
Sarangapani, J.: Neural Network Control of Nonlinear Discrete-Time Systems. CRC Press, Florida (2006)
Schottky, B., Saad, D.: Statistical Mechanics of EKF Learning in Neural Networks. J. Physics A 32(9), 1605–1621 (1999)
Shah, S., Palmieri, F., Datum, M.: Optimal Filtering Algorithms for Fast Learning in Feedforward Neural Networks. Neural Net. 5(5), 779–787 (1992)
Sum, J., Leung, C., Young, G.H., Kan, W.: On the Kalman Filtering Method in Neural Network Training and Pruning. IEEE Trans. Neural Netw. 10(1) (1999)
Yamada, T., Yabuta, T.: Dynamic System Identification Using Neural Networks. IEEE Trans. Systems, Man and Cyber. 23(1), 204–211 (1993)
Werbos, P.: Backpropagation Through Time: What It Does and How to Do It. Proc. IEEE 78(10), 1550–1560 (1990)
West, M., Harrison, J.: Bayesian Forecasting and Dynamic Linear Models. Springer, Heidelberg (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Arasaratnam, I., Haykin, S. (2008). Nonlinear Bayesian Filters for Training Recurrent Neural Networks. In: Gelbukh, A., Morales, E.F. (eds) MICAI 2008: Advances in Artificial Intelligence. MICAI 2008. Lecture Notes in Computer Science(), vol 5317. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88636-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-88636-5_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-88635-8
Online ISBN: 978-3-540-88636-5
eBook Packages: Computer ScienceComputer Science (R0)