Advertisement

Increasing Capacity of Association Memory by Means of Synaptic Clustering

  • Viacheslav OsaulenkoEmail author
  • Bernard Girau
  • Oleksandr Makarenko
  • Patrick Henaff
Article
  • 57 Downloads

Abstract

Making association is an essential property of human cognition. Systems that try to mimic this process and to make a coherent model of the world should have robust and high capacity association memory. Findings of nonlinear properties of dendritic tree suggest an alternative way how neurons can store associations. In this paper, we present a minimalistic neuron model with clustered synapses and show that it provides much higher association memory capacity compared to traditional models. Due to properties of sparse activation and tracking higher-order correlations in the input pattern an individual neuron can recognize thousands of patterns. Theoretical examination shows that this high capacity is reached because learning exact combinations of active neurons extends the dimension of an input space and thus increases pattern separability. We argue that such beneficial computational properties is realized in biological neural networks through synaptic clustering and sustaining sparse activity in memory-related areas.

Keywords

Association memory Dendritic nonlinearity Higher-order correlations Synaptic clustering 

Notes

Acknowledgements

This work is supported by Erasmus+ 2015-2017 and French embassy of Ukraine.

References

  1. 1.
    Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci 79(8):2554.  https://doi.org/10.1073/pnas.79.8.2554 MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Non-holographic associative memory. Nature 222(5197):960.  https://doi.org/10.1038/222960a0 CrossRefGoogle Scholar
  3. 3.
    Sommer FT, Wennekers T (2001) Associative memory in networks of spiking neurons. Neural Netw 14(6–7):825.  https://doi.org/10.1016/S0893-6080(01)00064-8 CrossRefGoogle Scholar
  4. 4.
    Zenke F, Agnes EJ, Gerstner W (2015) Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nature Commun 6:6922.  https://doi.org/10.1038/ncomms7922 CrossRefGoogle Scholar
  5. 5.
    Hunter R, Cutsuridis V, Cobb S, Graham B (2011) Improving associative memory in a model network of two- compartment spiking neurons, thesis submittedGoogle Scholar
  6. 6.
    Hebb (1949) DO the organization of behavior.  https://doi.org/10.2307/1418888. http://www.amazon.com/dp/0805843000
  7. 7.
    London M, Häusser M (2005) Dendritic Comput. Ann Rev Neurosci 28(1):503.  https://doi.org/10.1146/annurev.neuro.28.061604.135703 CrossRefGoogle Scholar
  8. 8.
    Stuart GJ, Spruston N (2015) Dendritic integration: 60 years of progress. Nature Neurosci 18(12):1713.  https://doi.org/10.1038/nn.4157 CrossRefGoogle Scholar
  9. 9.
    Du K, Wu YW, Lindroos R, Liu Y, Rózsa B, Katona G, Ding JB, Kotaleski JH (2017) Cell-typespecific inhibition of the dendritic plateau potential in striatal spiny projection neurons. Proc Natl Acad Sci 114(36):E7612.  https://doi.org/10.1073/pnas.1704893114 CrossRefGoogle Scholar
  10. 10.
    Mel BW, Schiller J, Poirazi P (2017) Synaptic plasticity in dendrites: complications and coping strategies. Curr Opin Neurobiol 43:177.  https://doi.org/10.1016/j.conb.2017.03.012 CrossRefGoogle Scholar
  11. 11.
    Branco T, Häusser M (2010) The single dendritic branch as a fundamental functional unit in the nervous system. Curr Opin Neurobiol 20(4):494.  https://doi.org/10.1016/j.conb.2010.07.009 CrossRefGoogle Scholar
  12. 12.
    Sjöström PJ, Rancz A, Roth A, Häusser M, Sjostrom PJ, Rancz EA, Roth A, Hausser M, PJ S, EA R, RA, HM, Sjostrom PJ, Rancz EA, Roth A, Hausser M, (2008) Dendritic excitability and synaptic plasticity. Physiol Rev 88(2):769.  https://doi.org/10.1152/physrev.00016.2007
  13. 13.
    Kastellakis G, Cai DJ, Mednick SC, Silva AJ, Poirazi P (2015) Synaptic clustering within dendrites: an emerging theory of memory formation. Prog Neurobiol 126:19.  https://doi.org/10.1016/j.pneurobio.2014.12.002 CrossRefGoogle Scholar
  14. 14.
    Bhalla UUS (2017) Dendrites, deep learning, and sequences in the hippocampus. Hippocampus 2014(6).  https://doi.org/10.1002/hipo.22806
  15. 15.
    Poirazi P, Mel BW (2001) Impact of active dendrites and structural plasticity on the memory capacity of neural tissue. Neuron 29(3):779.  https://doi.org/10.1016/S0896-6273(01)00252-5 CrossRefGoogle Scholar
  16. 16.
    Frank AC, Huang S, Zhou M, Gdalyahu A, Kastellakis G, Silva TK, Lu E, Wen X, Poirazi P, Trachtenberg JT, Silva AJ (2018) Hotspots of dendritic spine turnover facilitate clustered spine addition and learning and memory. Nat Commun 9(1):422.  https://doi.org/10.1038/s41467-017-02751-2 CrossRefGoogle Scholar
  17. 17.
    Bengio S, Bengio Y, Cloutier J (1995) On the search for new learning rules for ANNs. Neural Process Lett 2(4):26.  https://doi.org/10.1007/BF02279935 CrossRefGoogle Scholar
  18. 18.
    Brea J, Gerstner W (2016) Does computational neuroscience need new synaptic learning paradigms? Current opinion in behavioral sciences, pp 1–6.  https://doi.org/10.1016/j.cobeha.2016.05.012. http://linkinghub.elsevier.com/retrieve/pii/S2352154616301048
  19. 19.
    Tran-Van-Minh A, Cazé RD, Abrahamsson T, Cathala L, Gutkin BS, DiGregorio DA (2015) Contribution of sublinear and supralinear dendritic integration to neuronal computations. Front Cell Neurosci 9(March):1.  https://doi.org/10.3389/fncel.2015.00067 Google Scholar
  20. 20.
    Suvrathan A (2018) Beyond STDP towards diverse and functionally relevant plasticity rules. Curr Opin Neurobiol 54:12.  https://doi.org/10.1016/j.conb.2018.06.011 CrossRefGoogle Scholar
  21. 21.
    Mel BW, Koch C (1990) Sigma-pi learning: on radial basis functions and cortical associative learning. In: Advances in neural information processing systems. pp 474–481Google Scholar
  22. 22.
    Mel BW (1991) The clusteron: toward a simple abstraction for a complex neuron. Nips, pp 35–42Google Scholar
  23. 23.
    Plate TA (2000) Randomly connected sigma-pi neurons can form associator networks. Netw Comput Neural Syst 11:321–332.  https://doi.org/10.1088/0954-898X_11_4_305 CrossRefzbMATHGoogle Scholar
  24. 24.
    Baldi P (1987) Number of stable points for spin-glasses and neural networks of higher orders. Phys Rev Lett 58(9):913MathSciNetCrossRefGoogle Scholar
  25. 25.
    Giles CL, Maxwell T (1987) Learning, invariance, and generalization in high-order neural networks. Appl Opt 26:4972.  https://doi.org/10.1364/AO.26.004972 CrossRefGoogle Scholar
  26. 26.
    Krotov D, Hopfield JJ (2016) Dense associative memory for pattern recognition (Nips). arXiv:1606.01164
  27. 27.
    Guerguiev J, Lillicrap TP, Richards BA (2017) Towards deep learning with segregated dendrites. eLife 6:1.  https://doi.org/10.7554/eLife.22901 CrossRefGoogle Scholar
  28. 28.
    Wu X, Liu X, Li W, Wu Q (2018) Improved expressivity through dendritic neural networks. In: Advances in neural information processing systems. pp 8057–8068Google Scholar
  29. 29.
    Demircigil M, Heusel J, Löwe M, Upgang S, Vermet F (2017) On a model of associative memory with huge storage capacity. J Stat Phys 168(2):288.  https://doi.org/10.1007/s10955-017-1806-y MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Poirazi P, Mel B (2000) Memory capacity of linear vs . nonlinear models of dendritic integration. Advances in neural information processing systems (November), p 157Google Scholar
  31. 31.
    Sjöström PJ, Rancz a, Roth a, Häusser M (2010) Dendritic excitability and synaptic plasticity dendritic excitability and synaptic plasticity. Physiol Rev (November 2009) 1.  https://doi.org/10.1152/physrev.00016.2007
  32. 32.
    Poirazi P, Brannon T, Mel BW (2003) Pyramidal neuron as two-layered neural network. Neuron 37:989CrossRefGoogle Scholar
  33. 33.
    Hawkins J, Ahmad S (2016) Why neurons have thousands of synapses, a theory of sequence memory in neocortex. Front Neural Circuits 10(March):1.  https://doi.org/10.3389/fncir.2016.00023 Google Scholar
  34. 34.
    van Bommel B, Mikhaylova M (2016) Talking to the neighbours: the molecular and physiological mechanisms of clustered synaptic plasticity. Neurosci Biobeheav Rev 71:352.  https://doi.org/10.1016/j.neubiorev.2016.09.016 CrossRefGoogle Scholar
  35. 35.
    Olshausen BA, Field DJ (2004) Sparse coding of sensory inputs. Curr Opin Neurobiol 14(4):481.  https://doi.org/10.1016/j.conb.2004.07.007 CrossRefGoogle Scholar
  36. 36.
    Quiroga RQ, Kreiman G, Koch C, Fried I (2008) Sparse but not ’Grandmother-cell’ coding in the medial temporal lobe. Trends Cognit Sci 12(3):87.  https://doi.org/10.1016/j.tics.2007.12.003 CrossRefGoogle Scholar
  37. 37.
    Luczak A, McNaughton BL, Harris KD (2015) Packet-based communication in the cortex. Nat Rev Neurosci 16(12):745.  https://doi.org/10.1038/nrn4026 CrossRefGoogle Scholar
  38. 38.
    Gritsenko VI, Rachkovskij DA, Frolov AA, Gayler R, Kleyko D, Osipov E (2017) Neural distributed autoassociative memories: a survey. Cybern Comput Eng 2(188):5Google Scholar
  39. 39.
    Palm G (2013) Neural associative memories and sparse coding. Neural Netw 37:165.  https://doi.org/10.1016/j.neunet.2012.08.013 CrossRefGoogle Scholar
  40. 40.
    Fauth M, Tetzlaff C (2016) Opposing effects of neuronal activity on structural plasticity. Front Neuroanat 10(June):1.  https://doi.org/10.3389/fnana.2016.00075 Google Scholar
  41. 41.
    Knoblauch A (2005) Neural associative memory for brain modeling and information retrieval. Inf Process Lett 95(6 SPEC. ISS.):537.  https://doi.org/10.1016/j.ipl.2005.05.021 MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Knoblauch A, Palm G, Sommer FT (2010) Memory capacities for synaptic and structural plasticity. Neural Comput 22(2):289.  https://doi.org/10.1162/neco.2009.08-07-588 MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Graham B, Willshaw D (1997) Capacity and information efficiency of the associative net. Netw Comput Neural Syst 8(1):35.  https://doi.org/10.1088/0954-898X/8/1/005 CrossRefzbMATHGoogle Scholar
  44. 44.
    Markram H, Muller E et al (2015) Reconstruction and simulation of neocortical microcircuitry. Cell 163(2):456.  https://doi.org/10.1016/j.cell.2015.09.029 CrossRefGoogle Scholar
  45. 45.
    Lane SH, Handelman DA, Gelfand JJ (1992) Theory and development of higher-order CMAC neural networks. IEEE Control Syst 12(2):23.  https://doi.org/10.1109/37.126849 CrossRefGoogle Scholar
  46. 46.
    Taylor JG, Coombes S (1993) Learning higher order correlations. Neural Netw 6:423CrossRefGoogle Scholar
  47. 47.
    Chen D, Li S, Wu Q (2018) Rejecting chaotic disturbances using a super-exponential-zeroing neurodynamic approach for synchronization of chaotic sensor systems. Sensors 19(1):74.  https://doi.org/10.3390/s19010074 CrossRefGoogle Scholar
  48. 48.
    Ahmad S, Hawkins J (2015) Properties of sparse distributed representations and their application to hierarchical temporal memory.  https://doi.org/10.1063/1.4918346, arXiv:1503.07469
  49. 49.
    Ahmad S, Hawkins J (2016) How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites. arXiv:1601.00720 [q–bio.NC]
  50. 50.
    Kanerva P (1988) Sparse distributed memory. MIT Press, BostonzbMATHGoogle Scholar
  51. 51.
    Kanerva P (2009) Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cognit Comput 1(2):139.  https://doi.org/10.1007/s12559-009-9009-8 CrossRefGoogle Scholar
  52. 52.
    Plate TA (2012) Distributed representations. Encycl Cognit Sci 59:77Google Scholar
  53. 53.
    Rachkovskij DA, Kussul EM, Baidyk TN (2013) Building a world model with structure-sensitive sparse binary distributed representations. Biol Inspired Cognit Archit 3:64.  https://doi.org/10.1016/j.bica.2012.09.004 CrossRefGoogle Scholar
  54. 54.
    Knoblauch A, Sommer FT (2016) Structural plasticity, effectual connectivity, and memory in cortex. Front Neuroanat 10(June):1.  https://doi.org/10.3389/fnana.2016.00063 Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Igor Sikorsky Kyiv Polytechnic InstituteNational Technical University of UkraineKyivUkraine
  2. 2.LORIA UMR 7503 Université de Lorraine-INRIA-CNRSNancyFrance

Personalised recommendations