Analysis of the Dynamics of the Echo State Network Model Using Recurrence Plot
At the beginning of the 2000s, a specific type of Recurrent Neural Networks (RNNs) was developed with the name Echo State Network (ESN). The model has become popular during the last 15 years in the area of temporal learning. The model has a RNN (named reservoir) that projects an input sequence in a feature map. The reservoir has two main parameters that impact the accuracy of the model: the reservoir size (number of neurons in the RNN) and the spectral radius of the hidden-hidden recurrent weight matrix. In this article, we analyze the impact of these parameters using the Recurrence Plot technique, which is a useful tool for visualizing chaotic systems. Experiments carried out with three well-known dynamical systems show the relevance of the spectral radius in the reservoir projections.
KeywordsRecurrent Neural Network Echo State Network Recurrence Plot Chaotic systems Time-series problems
This work was supported by the Czech Science Foundation under the grant no. GJ16-25694Y, and by the projects SP2018/126 and SP2018/130 of the Student Grant System, VSB-Technical University of Ostrava, and it has been supported by the Czech Science Foundation (GAČR) under research project No. 18-18858S.
- 2.Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. German National Research Center for Information Technology, Technical report, 148 (2001)Google Scholar
- 7.Basterrech, S.: Empirical analysis of the necessary and sufficient conditions of the echo state property. In: 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, 14-19 May 2017, pp. 888–896 (2017)Google Scholar
- 10.Lukoševičius, M.: A Practical Guide to Applying Echo State Networks. In: Montavon, G., Orr, G., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science, vol. 7700, pp. 659–686. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_36CrossRefGoogle Scholar
- 15.Steil, J.J.: Backpropagation-Decorrelation: online recurrent learning with O(N) complexity. In: Proceedings of IJCNN 04, vol. 1 (2004)Google Scholar
- 19.Basterrech, S., Rubino, G.: Echo state queueing network: a new reservoir computing learning tool. In: 10th IEEE Consumer Communications and Networking Conference, CCNC 2013, Las Vegas, NV, USA, 11-14 January 2013, pp. 118–123 (2013). http://dx.doi.org/10.1109/CCNC.2013.6488435