Time Window Width Influence on Dynamic BPTT(h) Learning Algorithm Performances: Experimental Study
The purpose of the research addressed in this paper is to study the influence of the time window width in dynamic truncated BackPropagation Through Time BPTT(h) learning algorithms. Statistical experiments based on the identification of a real biped robot balancing mechanism are carried out to raise the link between the window width and the stability, the speed and the accuracy of the learning. The time window width choice is shown to be crucial for the convergence speed of the learning process and the generalization ability of the network. Although, a particular attention is brought to a divergence problem (gradient blow up) observed with the assumption where the net parameters are constant along the window. The limit of this assumption is demonstrated and parameters evolution storage, used as a solution for this problem, is detailed.
KeywordsLearning Rate Convergence Speed Recurrent Neural Network Window Width Generalization Ability
Unable to display preview. Download preview PDF.
- 2.Tsung, F.-S.: Modeling Dynamical Systems with Recurrent Neural Networks. PhD thesis, Department of Computer Science. University of California, San Diego (1994)Google Scholar
- 7.Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation Parallel distributed processing: explorations in the microstructure of cognition. In: Rumelhart, D.E., Mc- Clelland, J.L., the PDP Research Group (eds.), pp. 318–362. MIT Press, Cambridge (1986)Google Scholar
- 8.Williams, R.J., Zipser, D.: Gradient-based learning algorithms for recurrent connectionist networks. In: Chauvin, Y., Rumelhart, D.E. (eds.) Backpropagation: Theory, Architectures, and Applications, Erlbaum, Hillsdale, NJ (1990)Google Scholar
- 9.Williams, R.J., Peng, J.: An efficient gradient–based algorithm for on–line training of recurrent network trajectories. Neural Computation, vol. 2, pp. 490–501. MIT Press, Cambridge (1990)Google Scholar
- 10.Campolucci, P., Uncini, A., Piazza, F., Rao, B.D.: On-Line Learning Algorithms for Locally Recurrent Neural Networks. IEEE-NN 10, 253 (1999)Google Scholar