Improving Long- Term Online Prediction with Decoupled Extended Kalman Filters
Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform traditional RNNs when dealing with sequences involving not only short-term but also long-term dependencies. The decoupled extended Kalman filter learning algorithm (DEKF) works well in online environments and reduces significantly the number of training steps when compared to the standard gradient-descent algorithms. Previous work on LSTM, however, has always used a form of gradient descent and has not focused on true online situations. Here we combine LSTM with DEKF and show that this new hybrid improves upon the original learning algorithm when applied to online processing.
KeywordsGradient Descent Extended Kalman Filter Recurrent Neural Network Neural Computation Memory Block
Unable to display preview. Download preview PDF.
- 4.Haykin, S.: Neural networks: a comprehensive foundation. Prentice-Hall (1999).Google Scholar
- 5.Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J.: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. Kremer, S. C., Kolen, J. F. (eds.): A field guide to dynamical recurrent neural networks (2001). IEEE Press.Google Scholar