Significance-Based Pruning for Reservoir’s Neurons in Echo State Networks
Echo State Networks (ESNs) are a family of Recurrent Neural Networks (RNNs), that can be trained efficiently and robustly. Their main characteristic is the partitioning of the recurrent part of the network, the reservoir, from the non-recurrent part, the latter being the only component which is explicitly trained. To ensure good generalization capabilities, the reservoir is generally built from a large number of neurons, whose connectivity should be designed in a sparse pattern. Recently, we proposed an unsupervised online criterion for performing this sparsification process, based on the idea of significance of a synapse, i.e., an approximate measure of its importance in the network. In this paper, we extend our criterion to the direct pruning of neurons inside the reservoir, by defining the significance of a neuron in terms of the significance of its neighboring synapses. Our experimental validation shows that, by combining pruning of neurons and synapses, we are able to obtain an optimally sparse ESN in an efficient way. In addition, we briefly investigate the resulting reservoir’s topologies deriving from the application of our procedure.
KeywordsEcho State Networks Recurrent Neural Networks Pruning Least-Square
Unable to display preview. Download preview PDF.
- 2.Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks - with an erratum note. Tech. rep. (2001)Google Scholar
- 4.Newman, M.: Networks: an introduction. Oxford University Press (2010)Google Scholar
- 5.Scardapane, S., Nocco, G., Comminiello, D., Scarpiniti, M., Uncini, A.: An effective criterion for pruning reservoir’s connections in echo state networks. In: 2014 International Joint Conference in Neural Networks, pp. 1205–1212 (2014)Google Scholar