Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Neural Networks

ICANN 2014: Artificial Neural Networks and Machine Learning – ICANN 2014 pp 659–666Cite as

Self-generated Off-line Memory Reprocessing Strongly Improves Generalization in a Hierarchical Recurrent Neural Network

  • Conference paper

  • 4256 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 8681))

Abstract

Strong experimental evidence suggests that cortical memory traces are consolidated during off-line memory reprocessing that occurs in the off-line states of sleep or waking rest. It is unclear, what plasticity mechanisms are involved in this process and what changes are induced in the network in the off-line regime. Here, we examine a hierarchical recurrent neural network that performs unsupervised learning on natural face images of different persons. The proposed network is able to self-generate memory replay while it is decoupled from external stimuli. Remarkably, the recognition performance is tremendously boosted after this off-line regime specifically for the novel face views that were not shown during the initial learning. This effect is independent of synapse-specific plasticity, relying completely on homeostatic regulation of intrinsic excitability. Comparing a purely feed-forward network configuration with the full version reveals a substantially stronger boost in recognition performance for the fully recurrent network architecture after the off-line regime.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Jitsev, J., von der Malsburg, C.: Experience-driven formation of parts-based representations in a model of layered visual memory. Front. Comput. Neurosci. 3, 15 (2009)

    Article  Google Scholar 

  2. Lücke, J.: Receptive field self-organization in a model of the fine structure in V1 cortical columns. Neural Comput. 21(10), 2805–2845 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  3. Fries, P., Nikolić, D., Singer, W.: The gamma cycle. Trends Neurosci. 30(7), 309–316 (2007)

    Article  Google Scholar 

  4. Jitsev, J.: On the self-organization of a hierarchical memory for compositional object representation in the visual cortex. PhD thesis, Goethe University Frankfurt, Frankfurt Institute for Advanced Studies (November 2010)

    Google Scholar 

  5. Martinez, A.M., Benavente, R.: The AR face database. Technical Report 24, CVC Technical Report 24 (June 1998)

    Google Scholar 

  6. Jitsev, J., von der Malsburg, C.: Off-line memory reprocessing following on-line unsupervised learning strongly improves recognition performance in a hierarchical visual memory. In: International Joint Conference on Neural Networks (IJCNN), pp. 3123–3130. IEEE (July 2010)

    Google Scholar 

  7. Rasch, B., Born, J.: Maintaining memories by reactivation. Curr. Opin. Neurobiol. 17(6), 698–703 (2007)

    Article  Google Scholar 

  8. Diekelmann, S., Born, J.: The memory function of sleep. Nat. Rev. Neurosci. 11(2), 114–126 (2010)

    Google Scholar 

  9. Zhang, W., Linden, D.J.: The other side of the engram: experience-driven changes in neuronal intrinsic excitability. Nat. Rev. Neurosci. 4(11), 885–900 (2003)

    Article  Google Scholar 

  10. Karmarkar, U.R., Buonomano, D.V.: Different forms of homeostatic plasticity are engaged with distinct temporal profiles. Eur. J. Neurosci. 23(6), 1575–1584 (2006)

    Article  Google Scholar 

  11. Axmacher, N., Haupt, S., Fernández, G., Elger, C.E., Fell, J.: The role of sleep in declarative memory consolidation–direct evidence by intracranial EEG. Cereb. Cortex 18(3), 500–507 (2008)

    Article  Google Scholar 

  12. Lahl, O., Wispel, C., Willigens, B., Pietrowsky, R.: An ultra short episode of sleep is sufficient to promote declarative memory performance. J. Sleep Res. 17, 3–10 (2008)

    Article  Google Scholar 

  13. Tononi, G., Cirelli, C.: Sleep and synaptic homeostasis: A hypothesis. Brain Res. Bull. 62(2), 143–150 (2003)

    Article  Google Scholar 

  14. Olcese, U., Esser, S.K., Tononi, G.: Sleep and synaptic renormalization: A computational study. J. Neurophysiol. 104(6), 3476–3493 (2010)

    Article  Google Scholar 

  15. Tononi, G., Cirelli, C.: Sleep and the price of plasticity: from synaptic and cellular homeostasis to memory consolidation and integration. Neuron 81(1), 12–34 (2014)

    Article  Google Scholar 

  16. Crick, F., Mitchison, G.: The function of dream sleep. Nature 304(5922), 111–114 (1983)

    Article  Google Scholar 

  17. Baran, B., Wilson, J., Spencer, R.C.: REM-dependent repair of competitive memory suppression. Exp. Brain Res. 203(2), 471–477 (2010)

    Article  Google Scholar 

  18. Cirelli, C., Tononi, G.: Is sleep essential? PLoS Biol. 6(8), e216 (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Functional Neural Circuits Group, Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6), Forschungszentrum Juelich, 52425, Juelich, Germany

    Jenia Jitsev

Authors
  1. Jenia Jitsev
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Informatics, University of Hamburg, Vogt-Kölln-Straße 30, 22527, Hamburg, Germany

    Stefan Wermter , Cornelius Weber  & Sven Magg ,  & 

  2. Department of Informatics, Nicolaus Compernicus University, ul. Grudziądzka 5, 87-100, Torun, Poland

    Włodzisław Duch

  3. Department of Modern Languages, University of Helsinki, P.O. Box 24, 00014, Helsinki, Finland

    Timo Honkela

  4. Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Acad. G. Bonchev str. bl. 25A, 1113, Sofia, Bulgaria

    Petia Koprinkova-Hristova

  5. Institute of Neural Information Processing, University of Ulm, 89069, Oberer Eselsberg, Ulm, Germany

    Günther Palm

  6. Department of Information Systems, Quartier UNIL-Dorigny, Bâtiment Internef, University of Lausanne, 1015, Lausanne, Switzerland

    Alessandro E. P. Villa

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Jitsev, J. (2014). Self-generated Off-line Memory Reprocessing Strongly Improves Generalization in a Hierarchical Recurrent Neural Network. In: Wermter, S., et al. Artificial Neural Networks and Machine Learning – ICANN 2014. ICANN 2014. Lecture Notes in Computer Science, vol 8681. Springer, Cham. https://doi.org/10.1007/978-3-319-11179-7_83

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-11179-7_83

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-11178-0

  • Online ISBN: 978-3-319-11179-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation