Significance-Based Pruning for Reservoir’s Neurons in Echo State Networks

  • Simone Scardapane
  • Danilo Comminiello
  • Michele Scarpiniti
  • Aurelio Uncini

Abstract

Echo State Networks (ESNs) are a family of Recurrent Neural Networks (RNNs), that can be trained efficiently and robustly. Their main characteristic is the partitioning of the recurrent part of the network, the reservoir, from the non-recurrent part, the latter being the only component which is explicitly trained. To ensure good generalization capabilities, the reservoir is generally built from a large number of neurons, whose connectivity should be designed in a sparse pattern. Recently, we proposed an unsupervised online criterion for performing this sparsification process, based on the idea of significance of a synapse, i.e., an approximate measure of its importance in the network. In this paper, we extend our criterion to the direct pruning of neurons inside the reservoir, by defining the significance of a neuron in terms of the significance of its neighboring synapses. Our experimental validation shows that, by combining pruning of neurons and synapses, we are able to obtain an optimally sparse ESN in an efficient way. In addition, we briefly investigate the resulting reservoir’s topologies deriving from the application of our procedure.

Keywords

Echo State Networks Recurrent Neural Networks Pruning Least-Square 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Butcher, J., Verstraeten, D., Schrauwen, B., Day, C., Haycock, P.: Reservoir computing and extreme learning machines for non-linear time-series data analysis. Neural Networks 38, 76–89 (2013)CrossRefGoogle Scholar
  2. 2.
    Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks - with an erratum note. Tech. rep. (2001)Google Scholar
  3. 3.
    Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Computer Science Review 3(3), 127–149 (2009)MATHCrossRefGoogle Scholar
  4. 4.
    Newman, M.: Networks: an introduction. Oxford University Press (2010)Google Scholar
  5. 5.
    Scardapane, S., Nocco, G., Comminiello, D., Scarpiniti, M., Uncini, A.: An effective criterion for pruning reservoir’s connections in echo state networks. In: 2014 International Joint Conference in Neural Networks, pp. 1205–1212 (2014)Google Scholar
  6. 6.
    Siegelmann, H.T.: Neural and super-turing computing. Minds and Machines 13(1), 103–114 (2003)MATHCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Simone Scardapane
    • 1
  • Danilo Comminiello
    • 1
  • Michele Scarpiniti
    • 1
  • Aurelio Uncini
    • 1
  1. 1.Department of Information Engineering, Electronics and Telecommunications (DIET)“Sapienza” University of RomeRomeItaly

Personalised recommendations