Advertisement

Fast Spectral Radius Initialization for Recurrent Neural Networks

  • Claudio GallicchioEmail author
  • Alessio Micheli
  • Luca Pedrelli
Conference paper
Part of the Proceedings of the International Neural Networks Society book series (INNS, volume 1)

Abstract

In this paper we address the problem of grounded weights initialization for Recurrent Neural Networks. Specifically, we propose a method, rooted in the field of Random Matrix theory, to perform a fast initialization of recurrent weight matrices that meet specific constraints on their spectral radius. Focusing on the Reservoir Computing (RC) framework, the proposed approach allows us to overcome the typical computational bottleneck related to the eigendecomposition of large matrices, enabling to efficiently design large reservoir networks and hence to address time-series tasks characterized by medium/big datasets. Experimental results show that the proposed method enables an accurate control of the spectral radius of randomly initialized recurrent matrices, providing an initialization approach that is extremely more efficient compared to common RC practice.

Keywords

Recurrent Neural Networks Echo state networks Spectral radius of random matrices Time-series prediction 

References

  1. 1.
    Bai, Z., Yin, Y.: Limiting behavior of the norm of products of random matrices and two problems of Geman-Hwang. Probab. Theory Relat. Fields 73(4), 555–569 (1986)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bordenave, C., Caputo, P., Chafaï, D., Tikhomirov, K.: On the spectral radius of a random matrix. arXiv preprint arXiv:1607.05484 (2016)
  3. 3.
    Boulanger-Lewandowski, N., Bengio, Y., Vincent, P.: Modeling temporal dependencies in high-dimensional sequences: application to polyphonic music generation and transcription. In: Proceedings of the 29th International Conference on Machine Learning (2012)Google Scholar
  4. 4.
    Gallicchio, C., Micheli, A.: Architectural and Markovian factors of echo state networks. Neural Netw. 24(5), 440–456 (2011)CrossRefGoogle Scholar
  5. 5.
    Gallicchio, C., Micheli, A., Pedrelli, L.: Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017)CrossRefGoogle Scholar
  6. 6.
    Gallicchio, C., Micheli, A., Pedrelli, L.: Design of deep echo state networks. Neural Netw. 108, 33–47 (2018)CrossRefGoogle Scholar
  7. 7.
    Gallicchio, C., Micheli, A., Pedrelli, L.: Comparison between DeepESNs and gated RNNs on multivariate time-series prediction. In: Proceedings of the 27th European Symposium on Artificial Neural Networks (ESANN) (in press)Google Scholar
  8. 8.
    Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)CrossRefGoogle Scholar
  9. 9.
    Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)CrossRefGoogle Scholar
  10. 10.
    Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: Proceedings of International Conference on Machine Learning (ICML), vol. 28, pp. 1310–1318 (2013)Google Scholar
  11. 11.
    Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147 (2013)Google Scholar
  12. 12.
    Tao, T., Vu, V., Krishnapur, M., et al.: Random matrices: universality of ESDs and the circular law. Ann. Probab. 38(5), 2023–2065 (2010)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Claudio Gallicchio
    • 1
    Email author
  • Alessio Micheli
    • 1
  • Luca Pedrelli
    • 1
  1. 1.Department of Computer ScienceUniversity of PisaPisaItaly

Personalised recommendations