Skip to main content
SpringerLink
Log in

Hebbian learning from higher-order correlations requires crosstalk minimization

  • Original Paper
  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

Activity-dependent synaptic plasticity should be extremely connection specific, though experiments have shown it is not, and biophysics suggests it cannot be. Extreme specificity (near-zero “crosstalk”) might be essential for unsupervised learning from higher-order correlations, especially when a neuron has many inputs. It is well known that a normalized nonlinear Hebbian rule can learn “unmixing” weights from inputs generated by linearly combining independently fluctuating nonGaussian sources using an orthogonal mixing matrix. We previously reported that even if the matrix is only approximately orthogonal, a nonlinear-specific Hebbian rule can usually learn almost correct unmixing weights (Cox and Adams in Front Comput Neurosci 3: doi:10.3389/neuro.10.011.2009 2009). We also reported simulations that showed that as crosstalk increases from zero, the learned weight vector first moves slightly away from the crosstalk-free direction and then, at a sharp threshold level of inspecificity, jumps to a completely incorrect direction. Here, we report further numerical experiments that show that above this threshold, residual learning is driven instead almost entirely by second-order input correlations, as occurs using purely Gaussian sources or a linear rule, and any amount of crosstalk. Thus, in this “ICA” model learning from higher-order correlations, required for unmixing, requires high specificity. We compare our results with a recent mathematical analysis of the effect of crosstalk for exactly orthogonal mixing, which revealed that a second, even lower, threshold, exists below which successful learning is impossible unless weights happen to start close to the correct direction. Our simulations show that this also holds when the mixing is not exactly orthogonal. These results suggest that if the brain uses simple Hebbian learning, it must operate with extraordinarily accurate synaptic plasticity to ensure powerful high-dimensional learning. Synaptic crowding would preclude this when inputs are numerous, and we propose that the neocortex might be distinguished by special circuitry that promotes extreme specificity for high-dimensional nonlinear learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Adams P, Cox K (2002) A new interpretation of thalamocortical circuitry. Philos Trans R Soc Lond Ser B Biol Sci 357(1428):1767–1779. doi:10.1098/rstb.2002.1164

    Article  Google Scholar 

  • Adams PR, Cox KJA (2002b) Synaptic Darwinism and neocortical function. Neurocomputing 42(1–4):197–214. doi:10.1016/s0925-2312(01)00591-4

    Article  Google Scholar 

  • Adams PR, Cox KJA (2006) A neurobiological perspective on building intelligent devices. Neuromorphic Eng 3(1):2–8

    Google Scholar 

  • Amari S-I (1998) Natural gradient works efficiently in learning. Neural Comput 10(2):251–276. doi:10.1162/089976698300017746

    Article  Google Scholar 

  • Amari S-I, Chen T-P, Cichocki A (1997) Stability analysis of learning algorithms for blind source separation. Neural Netw 10(8):1345–1351. doi:10.1016/s0893-6080(97)00039-7

    Article  PubMed  Google Scholar 

  • Amari S, Cichocki A, Yang HH (1996) A new learning algorithm for blind signal separation. In: Touretzky D, Mozer M, Hasselmo M (eds) Advances in neural information processing systems. MIT Press, Cambridge, pp 757–763. doi:10.1016/s0893-6080(97)

  • Araya R, Jiang J, Eisenthal KB, Yuste R (2006) The spine neck filters membrane potentials. Proc Natl Acad Sci USA 103(47):17961–17966. doi:10.1073/pnas.0608755103

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Atick JJ, Redlich AN (1990) Towards a theory of early visual processing. Neural Comput 2(3):308–320. doi:10.1162/neco.1990.2.3.308

    Article  Google Scholar 

  • Banitt Y, Martin K, Segev I (2007) A biologically realistic model of contrast invariant orientation tuning by thalamocortical synaptic depression. J Neurosci 27(38):10230–10239. doi:10.1523/jneurosci.1640-7

  • Bell AJ, Sejnowski TJ (1997) The “independent components” of natural scenes are edge filters. Vis Res 37(23):3327–3338

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Bi GQ (2002) Spatiotemporal specificity of synaptic plasticity: cellular rules and mechanisms. Biol Cybern 87(5–6):319–332. doi:10.1007/s00422-002-0349-7

    PubMed  Google Scholar 

  • Bonhoeffer T, Staiger V, Aertsen A (1989) Synaptic plasticity in rat hippocampal slice cultures: local “Hebbian” conjunction of pre- and postsynaptic stimulation leads to distributed synaptic enhancement. Proc Natl Acad Sci U S A 86(20):8113–8117

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Botelho F, Jamison JE (2004) Qualitative behavior of differential equations associated with artificial neural networks. J. Dyn. Diff. Equ. 16(1):179–204. doi:10.1023/B:JODY.0000041285.36221.bf

    Article  Google Scholar 

  • Chen S, Gopinath R (2000) Gaussianization. In: Advances in Neural Information Processing Systems, pp 423–429

  • Cox KJA, Adams P (2009) Hebbian crosstalk prevents nonlinear unsupervised learning. Front. Comput. Neurosci. 3: doi:10.3389/neuro.10.011.2009

  • Cox KJA, Adams PR (2012) From life to mind: 2 prosaic miracles? In: Simeonov PL, Smith LS, Ehresmann AC (eds) Integral biomathics: tracing the road to reality. Proceedings of iBioMath 2011, Paris and ACIB’11. Springer, Stirling

  • Da Costa N, Martin K (2011) How thalamus connects to spiny stellate cells in the cat’s visual cortex. J Neurosci 31(8):2925–2937

    Article  PubMed  Google Scholar 

  • Eigen M (1971a) Molecular self-organization and the early stages of evolution. Experientia 27(11):149–212

    CAS  PubMed  Google Scholar 

  • Eigen M (1971b) Selforganization of matter and the evolution of biological macromolecules. Naturwissenschaften 58(10):465– 523

  • Elliott T (2012) Cross-talk induces bifurcations in nonlinear models of synaptic plasticity. Neural Comput. 24:1–68

    Article  Google Scholar 

  • Engert F, Bonhoeffer T (1997) Synapse specificity of long-term potentiation breaks down at short distances. Nature 388(6639):279–284. doi:10.1038/40870

    Article  CAS  PubMed  Google Scholar 

  • Feng D, Marshburn D, Jen D, Weinberg RJ, Taylor RM 2nd, Burette A (2007) Stepping into the third dimension. J Neurosci 27(47):12757–12760. doi:10.1523/JNEUROSCI.2846-07.2007

    Article  CAS  PubMed  Google Scholar 

  • Field DJ (1994) What is the goal of sensory coding? Neural Comput 6(4):559–601. doi:10.1162/neco.1994.6.4.559

    Article  Google Scholar 

  • Foldiak P (1990) Forming sparse representations by local anti-Hebbian learning. Biol Cybern 64(2):165–170

    CAS  PubMed  Google Scholar 

  • Friedman JH (1987) Exploratory projection pursuit. J Am Stat Assoc 82(397):249–266

    Article  Google Scholar 

  • Harvey CD, Svoboda K (2007) Locally dynamic synaptic learning rules in pyramidal neuron dendrites. Nature 450(7173):1195–1200. doi:10.1038/nature06416

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Hinton GE, Sejnowski TJ (1999) Unsupervised learning: foundations of neural computation. MIT Press, Cambridge

    Google Scholar 

  • Hoyer PO, Hyvarinen A (2000) Independent component analysis applied to feature extraction from colour and stereo images. Network 11(3):191–210

    Article  CAS  PubMed  Google Scholar 

  • Hyvärinen A (2013) Independent component analysis: recent advances. Philos Trans R Soc A Math Phys Eng Sci 371(1984). doi:10.1098/rsta.2011.0534

  • Hyvarinen A, Hoyer P (2000) Emergence of phase- and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural Comput. 12(7):1705–1720

    Article  CAS  PubMed  Google Scholar 

  • Hyvarinen A, Hurri J, Hoyer P (2009) Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Springer Publishing Company, Incorporated

  • Hyvarinen A, Karhunen J, Oja E (2001) Independent component analysis. Wiley. New York. Available via http://worldcat.org. http://www.myilibrary.com?id=26480

  • Hyvärinen A, Oja E (1998) Independent component analysis by general nonlinear Hebbian-like learning rules. Signal Process 64(3):301–313. doi:10.1016/s0165-1684(97)00197-7

    Article  Google Scholar 

  • Jia H, Rochefort NL, Chen X, Konnerth A (2010) Dendritic organization of sensory input to cortical neurons in vivo. Nature 464(7293):1307–1312. doi:10.1038/nature08947

    Article  CAS  PubMed  Google Scholar 

  • Kim K-H, Gaba S, Wheeler D, Cruz-Albrecht JM, Hussain T, Srinivasa N, Lu W (2011) A functional hybrid memristor crossbar-array/CMOS system for data storage and neuromorphic applications. Nano Lett 12(1):389–395. doi:10.1021/nl203687n

    Article  PubMed  Google Scholar 

  • Koch C, Zador A (1993) The function of dendritic spines: devices subserving biochemical rather than electrical compartmentalization. J Neurosci 13(2):413–422

    CAS  PubMed  Google Scholar 

  • Kuang X, Poletti M, Victor JD, Rucci M (2012) Temporal encoding of spatial information during active visual fixation. Curr Biol CB 22(6):510–514. doi:10.1016/j.cub.2012.01.050

    Article  CAS  Google Scholar 

  • Likharev KK (2008) Defect-Tolerant Hybrid CMOS/Nanoelectronic Circuits. Paper presented at the Proceedings of the 2008 IEEE International Symposium on Defect and Fault Tolerance of VLSI Systems

  • Matsuzaki M, Honkura N, Ellis-Davies GC, Kasai H (2004) Structural basis of long-term potentiation in single dendritic spines. Nature 429(6993):761–766. doi:10.1038/nature02617

    Article  CAS  PubMed  Google Scholar 

  • Noguchi J, Matsuzaki M, Ellis-Davies GC, Kasai H (2005) Spine-neck geometry determines NMDA receptor-dependent Ca2+ signaling in dendrites. Neuron 46(4):609–622. doi:10.1016/j.neuron.2005.03.015

    Article  CAS  PubMed  Google Scholar 

  • Oja E (1982) A simplified neuron model as a principal component analyzer. J Math Biol 15(3):267–273

    Article  CAS  PubMed  Google Scholar 

  • Palmer LM, Stuart GJ (2009) Membrane potential changes in dendritic spines during action potentials and synaptic input. J Neurosci 29(21):6897–6903. doi:10.1523/JNEUROSCI.5847-08.2009

    Article  CAS  PubMed  Google Scholar 

  • Radulescu A, Cox K, Adams P (2009) Hebbian errors in learning: an analysis using the Oja model. J Theor Biol 258(4):489–501. doi:10.1016/j.jtbi.2009.01.036

    Article  PubMed  Google Scholar 

  • Radulescu A, Adams P (2013) Hebbian crosstalk and input segregation. J Theor Biol (337)133–149. doi:10.1016/j.jtbi.2013.08.004

  • Rattray M (2002) Stochastic trapping in a solvable model of on-line independent component analysis. Neural Comput 14(2):421–435. doi:10.1162/08997660252741185

    Article  PubMed  Google Scholar 

  • Reynolds T, Hartell NA (2000) An evaluation of the synapse specificity of long-term depression induced in rat cerebellar slices. J Physiol 527(Pt 3):563–577

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Sabatini BL, Oertner TG, Svoboda K (2002) The life cycle of Ca(2+) ions in dendritic spines. Neuron 33(3):439–452

    Article  CAS  PubMed  Google Scholar 

  • Scheffer M, Bascompte J, Brock WA, Brovkin V, Carpenter SR, Dakos V, Held H, van Nes EH, Rietkerk M, Sugihara G (2009) Early-warning signals for critical transitions. Nature 461(7260):53–59. doi:10.1038/nature08227

    Article  CAS  PubMed  Google Scholar 

  • Schuman EM, Madison DV (1994) Locally distributed synaptic potentiation in the hippocampus. Science 263(5146):532–536

    Article  CAS  PubMed  Google Scholar 

  • Shan H, Zhang L, Cottrell GW (2007) Recursive ICA. Adv Neural Inf Process Syst 19:1273–1280

    Google Scholar 

  • Somers DC, Nelson SB, Sur M (1995) An emergent model of orientation selectivity in cat visual cortical simple cells. J Neurosci 15(8):5448–5465

    CAS  PubMed  Google Scholar 

  • Srinivasan MV, Laughlin SB, Dubs A (1982) Predictive coding: a fresh view of inhibition in the retina. Proc R Soc Lond B Biol Sci 216(1205):427–459

    Article  CAS  PubMed  Google Scholar 

  • Strogatz SH (1994) Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. Westview Press, Boulder

    Google Scholar 

  • Triesch J (2007) Synergies between intrinsic and synaptic plasticity mechanisms. Neural Comput 19(4):885–909. doi:10.1162/neco.2007.19.4.885

    Article  PubMed  Google Scholar 

  • Vontobel PO, Robinett W, Kuekes PJ, Stewart DR, Straznicky J, Stanley Williams R (2009) Writing to and reading from a nano-scale crossbar memory based on memristors. Nanotechnology 20(42):425204. doi:10.1088/0957-4484/20/42/425204

    Article  PubMed  Google Scholar 

  • Wickens J (1988) Electrically coupled but chemically isolated synapses: dendritic spines and calcium in a rule for synaptic modification. Prog Neurobiol 31(6):507–528

    Article  CAS  PubMed  Google Scholar 

  • Wilbrecht L, Holtmaat A, Wright N, Fox K, Svoboda K (2010) Structural plasticity underlies experience-dependent functional plasticity of cortical circuits. J Neurosci 30(14):4927–4932. doi:10.1523/JNEUROSCI.6403-09.2010

  • Xu H-T, Pan F, Yang G, Gan W-B (2007) Choice of cranial window type for in vivo imaging affects dendritic spine turnover in the cortex. Nat Neurosci 10(5):549–551. http://www.nature.com/neuro/journal/v10/n5/suppinfo/nn1883_S1.html

  • Yuste R, Denk W (1995) Dendritic spines as basic functional units of neuronal integration. Nature 375(6533):682–684. doi:10.1038/375682a0

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

We are deeply grateful to Terry Elliott for extensive advice, helpful criticism and for providing advance copies, and patient explanations, of his analysis of crosstalk models. We also thank him for detailed, pointed and constructive criticism of earlier drafts of this paper, which he would have written quite differently, and for extensive discussions and dissections of Foldiak bars. We also thank Giancarlo La Camera and Luca Mazzucato for helpful reading of a draft of our paper.

Author information

Authors and Affiliations

  1. Department of Neurobiology and Behavior, Stony Brook University, Stony Brook, NY , 11794–5230, USA

    K. J. A. Cox & P. R. Adams

Authors
  1. K. J. A. Cox
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. R. Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. J. A. Cox.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cox, K.J.A., Adams, P.R. Hebbian learning from higher-order correlations requires crosstalk minimization. Biol Cybern 108, 405–422 (2014). https://doi.org/10.1007/s00422-014-0608-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-014-0608-4

Keywords

Search

Navigation