Use of a Sparse Structure to Improve Learning Performance of Recurrent Neural Networks
The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.
KeywordsRecurrent Neural Networks Sparse Structure Sparse Coding
Unable to display preview. Download preview PDF.
- 5.Kanerva, P.: Sparse distributed memory and related models, pp. 50–76. Oxford University Press, Inc. (1993)Google Scholar
- 6.Palm, G., Sommer, F.: Associative data storage and retrieval in neural networks. Models of Neural Networks III, 79–118 (1996)Google Scholar
- 8.Andreea, L., Pipa Gordon, T.J.: SORN: a self-organizing recurrent neural network. Front. Comput. Neurosci. 3(23) (2009), doi:10.3389/neuro.10.023Google Scholar
- 9.Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation, pp. 318–362. MIT Press (1986)Google Scholar
- 11.Hinoshita, W., Arie, H., Tani, J., Okuno, H.G., Ogata, T.: Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network. Neural Networks (in Press, 2011), doi:10.1016/j.neunet.2010.12.006Google Scholar
- 12.The ‘independent components’ of natural scenes are edge filters. Vision Research 37, 3327–3338 (1997)Google Scholar