Fast Spectral Radius Initialization for Recurrent Neural Networks
In this paper we address the problem of grounded weights initialization for Recurrent Neural Networks. Specifically, we propose a method, rooted in the field of Random Matrix theory, to perform a fast initialization of recurrent weight matrices that meet specific constraints on their spectral radius. Focusing on the Reservoir Computing (RC) framework, the proposed approach allows us to overcome the typical computational bottleneck related to the eigendecomposition of large matrices, enabling to efficiently design large reservoir networks and hence to address time-series tasks characterized by medium/big datasets. Experimental results show that the proposed method enables an accurate control of the spectral radius of randomly initialized recurrent matrices, providing an initialization approach that is extremely more efficient compared to common RC practice.
KeywordsRecurrent Neural Networks Echo state networks Spectral radius of random matrices Time-series prediction
- 2.Bordenave, C., Caputo, P., Chafaï, D., Tikhomirov, K.: On the spectral radius of a random matrix. arXiv preprint arXiv:1607.05484 (2016)
- 3.Boulanger-Lewandowski, N., Bengio, Y., Vincent, P.: Modeling temporal dependencies in high-dimensional sequences: application to polyphonic music generation and transcription. In: Proceedings of the 29th International Conference on Machine Learning (2012)Google Scholar
- 7.Gallicchio, C., Micheli, A., Pedrelli, L.: Comparison between DeepESNs and gated RNNs on multivariate time-series prediction. In: Proceedings of the 27th European Symposium on Artificial Neural Networks (ESANN) (in press)Google Scholar
- 10.Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: Proceedings of International Conference on Machine Learning (ICML), vol. 28, pp. 1310–1318 (2013)Google Scholar
- 11.Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147 (2013)Google Scholar