Advertisement

Use of a Sparse Structure to Improve Learning Performance of Recurrent Neural Networks

  • Hiromitsu Awano
  • Shun Nishide
  • Hiroaki Arie
  • Jun Tani
  • Toru Takahashi
  • Hiroshi G. Okuno
  • Tetsuya Ogata
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7064)

Abstract

The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.

Keywords

Recurrent Neural Networks Sparse Structure Sparse Coding 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Olshausen, B.A., Field, D.J.: Sparse coding of sensory inputs. Current Opinion in Neurobiology 14(4), 481–487 (2004)CrossRefGoogle Scholar
  2. 2.
    Waydo, S., Kraskov, A., Quian Quiroga, R., Fried, I., Koch, C.: Sparse representation in the human medial temporal lobe. J. Neurosci. 26(40), 10232–10234 (2006)CrossRefGoogle Scholar
  3. 3.
    Smith, E., Lewicki, M.: Efficient auditory coding. Nature 439(7079), 978–982 (2006)CrossRefGoogle Scholar
  4. 4.
    Vinje, W.E., Gallant, J.L.: Sparse Coding and Decorrelation in Primary Visual Cortex During Natural Vision. Science 287(5456), 1273–1276 (2000)CrossRefGoogle Scholar
  5. 5.
    Kanerva, P.: Sparse distributed memory and related models, pp. 50–76. Oxford University Press, Inc. (1993)Google Scholar
  6. 6.
    Palm, G., Sommer, F.: Associative data storage and retrieval in neural networks. Models of Neural Networks III, 79–118 (1996)Google Scholar
  7. 7.
    Jaeger, H., Haas, H.: Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication. Science 304(5667), 78–80 (2004)CrossRefGoogle Scholar
  8. 8.
    Andreea, L., Pipa Gordon, T.J.: SORN: a self-organizing recurrent neural network. Front. Comput. Neurosci. 3(23) (2009), doi:10.3389/neuro.10.023Google Scholar
  9. 9.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation, pp. 318–362. MIT Press (1986)Google Scholar
  10. 10.
    Yamashita, Y., Tani, J.: Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment. PLoS Comput. Biol. 4(11), e1000220 (2008)CrossRefGoogle Scholar
  11. 11.
    Hinoshita, W., Arie, H., Tani, J., Okuno, H.G., Ogata, T.: Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network. Neural Networks (in Press, 2011), doi:10.1016/j.neunet.2010.12.006Google Scholar
  12. 12.
    The ‘independent components’ of natural scenes are edge filters. Vision Research 37, 3327–3338 (1997)Google Scholar
  13. 13.
    Willmore, B., Tolhurst, D.: Characterizing the sparseness of neural codes. Network: Computation in Neural Systems 12(3), 255–270 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Hiromitsu Awano
    • 1
  • Shun Nishide
    • 1
  • Hiroaki Arie
    • 2
  • Jun Tani
    • 2
  • Toru Takahashi
    • 1
  • Hiroshi G. Okuno
    • 1
  • Tetsuya Ogata
    • 1
  1. 1.Graduate School of InformaticsKyoto UniversityKyotoJapan
  2. 2.RIKEN, Brain Science InstituteWako City, SaitamaJapan

Personalised recommendations