Abstract
In order to comprehend and enhance models that describes various brain regions it is important to study the dynamics of trained recurrent neural networks. Including Dale’s law in such models usually presents several challenges. However, this is an important aspect that allows computational models to better capture the characteristics of the brain. Here we present a framework to train networks using such constraint. Then we have used it to train them in simple decision making tasks. We characterized the eigenvalue distributions of the recurrent weight matrices of such networks. Interestingly, we discovered that the non-dominant eigenvalues of the recurrent weight matrix are distributed in a circle with a radius less than 1 for those whose initial condition before training was random normal and in a ring for those whose initial condition was random orthogonal. In both cases, the radius does not depend on the fraction of excitatory and inhibitory units nor the size of the network. Diminution of the radius, compared to networks trained without the constraint, has implications on the activity and dynamics that we discussed here.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/, software available from tensorflow.org
Anton H, Rorres C (2014) Elementary linear algebra. Wiley, Applications Version
Arnold VI (1992) Ordinary differential equations. Springer, Berlin, Heidelberg
Barak O (2017) Recurrent neural networks as versatile tools of neuroscience research. Curr Opin Neurobiol 46:1–6. https://doi.org/10.1016/j.conb.2017.06.003
Barranca VJ, Bhuiyan A, Sundgren M, Xing F (2022) Functional implications of dale’s law in balanced neuronal network dynamics and decision making. Front Neurosci. https://doi.org/10.3389/fnins.2022.801847
Bi Z, Zhou C (2020) Understanding the computation of time using neural network models. Proc Natl Acad Sci 117(19):10530–10540. https://doi.org/10.1073/pnas.1921609117
Bondanelli G, Ostojic S (2020) Coding with transient trajectories in recurrent neural networks. PLoS Comput Biol 16(2):1–36. https://doi.org/10.1371/journal.pcbi.1007655
Chollet F, et al (2015) Keras. https://keras.io
Churchland MM, Cunningham JP, Kaufman MT, Foster JD, Nuyujukian P, Ryu SI, Shenoy KV (2012) Neural population dynamics during reaching. Nature 487(7405):51–56. https://doi.org/10.1038/nature11129
Cornford J, Kalajdzievski D, Leite M, Lamarquette A, Kullmann DM, Richards B (2021) Learning to live with dale’s principle: Anns with separate excitatory and inhibitory units. bioRxiv https://doi.org/10.1101/2020.11.02.364968
Dale H (1935) Pharmacology and nerve-endings. Proc R Soc Med 28(3):319–332. https://doi.org/10.1177/003591573502800330
Freedman DJ, Assad JA (2006) Experience-dependent representation of visual categories in parietal cortex. Nature 443(7107):85–88. https://doi.org/10.1038/nature05078
Freeman W (2000) How brains make up their minds. Maps of the mind series, Phoenix, https://books.google.com.ar/books?id=a_GuQgAACAAJ
Funahashi K (1989) On the approximate realization of continuous mappings by neural networks. Neural Netw 2(3):183–192. https://doi.org/10.1016/0893-6080(89)90003-8
Funahashi K, Nakamura Y (1993) Approximation of dynamical systems by continuous time recurrent neural networks. Neural Netw 6(6):801–806. https://doi.org/10.1016/S0893-6080(05)80125-X
García del Molino LC, Pakdaman K, Touboul J, Wainrib G (2013) Synchronization in random balanced networks. Phys Rev E 88:042824. https://doi.org/10.1103/PhysRevE.88.042824
Golub GH, Loan CFV (2013) Matrix Comput. The Johns Hopkins University Press
Henaff M, Szlam A, LeCun Y (2016) Recurrent orthogonal networks and long-memory tasks. In: Balcan MF, Weinberger KQ (eds) Proceedings of The 33rd International Conference on Machine Learning, PMLR, New York, New York, USA, Proceedings of Machine Learning Research, vol 48, pp 2034–2042, https://proceedings.mlr.press/v48/henaff16.html
Hopfield JJ (1984) Neurons with graded response have collective computational properties like those of two-state neurons. Proc Natl Acad Sci 81(10):3088–3092. https://doi.org/10.1073/pnas.81.10.3088
Ingrosso A (2021) Optimal learning with excitatory and inhibitory synapses. PLoS Comput Biol 16(12):1–24. https://doi.org/10.1371/journal.pcbi.1008536
Ingrosso A, Abbott LF (2019) Training dynamically balanced excitatory-inhibitory networks. PLoS ONE 14(8):1–18. https://doi.org/10.1371/journal.pone.0220547
Jarne C (2021) Multitasking in RNN: an analysis exploring the combination of simple tasks. J Phys Complex 2(1):015009. https://doi.org/10.1088/2632-072x/abdee3
Jarne C (2022) Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks. Cogn Neurodyn. https://doi.org/10.1007/s11571-022-09802-5
Jarne C, Laje R (2020) Graceful degradation of recurrent neural networks as a function of network size, memory length, and connectivity damage. arXiv:https://arxiv.org/abs/1906.01094
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. CoRR abs/1412.6980, arXiv:http://arxiv.org/abs/1412.6980,
Le QV, Jaitly N, Hinton GE (2015) A simple way to initialize recurrent networks of rectified linear units. https://doi.org/10.48550/ARXIV.1504.00941, arxiv: https://arxiv.org/abs/1504.00941
Ludwig M, Leng G (2006) Dendritic peptide release and peptide-dependent behaviours. Nature Rev Neurosci 7(2):126–136. https://doi.org/10.1038/nrn1845
Mante V, Sussillo D, Shenoy KV, Newsome WT (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503(7474):78–84. https://doi.org/10.1038/nature12742
Murphy BK, Miller KD (2009) Balanced amplification: A new mechanism of selective amplification of neural activity patterns. Neuron 61(4):635–648. https://doi.org/10.1016/j.neuron.2009.02.005
Parisien C, Anderson CH, Eliasmith C (2008) Solving the problem of negative synaptic weights in cortical models. Neural Comput 20(6):1473–1494. https://doi.org/10.1162/neco.2008.07-06-295
Polyanin AD, Zaitsev VF (2017) Handbook of ordinary differential equations. Chapman and Hall
Rajakumar A, Rinzel J, Chen ZS (2021) Stimulus-driven and spontaneous dynamics in excitatory-inhibitory recurrent neural networks for sequence representation. Neural Comput 33(10):2603–2645. https://doi.org/10.1162/neco_a_01418
Rajan K, Abbott LF (2006) Eigenvalue spectra of random matrices for neural networks. Phys Rev Lett 97:188104. https://doi.org/10.1103/PhysRevLett.97.188104
Remington ED, Narain D, Hosseini EA, Jazayeri M (2018) Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron 98(5):1005-1019.e5. https://doi.org/10.1016/j.neuron.2018.05.020
Roitman JD, Shadlen MN (2002) Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J Neurosci 22(21):9475–9489. https://doi.org/10.1523/JNEUROSCI.22-21-09475.2002
Russo AA, Bittner SR, Perkins SM, Seely JS, London BM, Lara AH, Miri A, Marshall NJ, Kohn A, Jessell TM, Abbott LF, Cunningham JP, Churchland MM (2018) Motor cortex embeds muscle-like commands in an untangled population response. Neuron. https://doi.org/10.1016/j.neuron.2018.01.004
Russo AA, Khajeh R, Bittner SR, Perkins SM, Cunningham JP, Abbott L, Churchland MM (2020) Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation. Neuron 107(4):745–758
Song HF, Yang GR, Wang XJ (2016) Training excitatory-inhibitory recurrent neural networks for cognitive tasks: A simple and flexible framework. PLoS Comput Biol 12(2):1–30. https://doi.org/10.1371/journal.pcbi.1004792
Susman L, Mastrogiuseppe F, Brenner N, Barak O (2021) Quality of internal representation shapes learning performance in feedback neural networks. Phys Rev Res 3:013176. https://doi.org/10.1103/PhysRevResearch.3.013176
Sussillo D, Barak O (2013) Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput 25(3):626–649. https://doi.org/10.1162/NECO_a_00409
Sussillo D, Churchland MM, Kaufman MT, Shenoy KV (2015) A neural network that finds a naturalistic solution for the production of muscle activity. Nature Neurosci 18(7):1025–1033. https://doi.org/10.1038/nn.4042
Svensson E, Apergis-Schoute J, Burnstock G, Nusbaum MP, Parker D, Schiöth HB (2019) General principles of neuronal co-transmission: Insights from multiple model systems. Front Neural Circ. https://doi.org/10.3389/fncir.2018.00117
Vorontsov E, Trabelsi C, Kadoury S, Pal C (2017) On orthogonality and learning recurrent networks with long term dependencies. CoRR arXiv:abs/1702.00071
Vyas S, Golub MD, Sussillo D, Shenoy KV (2020) Computation through neural population dynamics. Ann Rev Neurosci 43(1):249–275. https://doi.org/10.1146/annurev-neuro-092619-094115
Williams AH, Kim TH, Wang F, Vyas S, Ryu SI, Shenoy KV, Schnitzer M, Kolda TG, Ganguli S (2018) Unsupervised discovery of demixed, low-dimensional neural dynamics across multiple timescales through tensor component analysis. Neuron 98(6):1099–1115. https://doi.org/10.1016/j.neuron.2018.05.015
Zhang X, Liu S, Chen ZS (2021) A geometric framework for understanding dynamic information integration in context-dependent computation. iScience. https://doi.org/10.1016/j.isci.2021.102919
Zhou Q, Jin T, Zhao H (2009) Correlation between eigenvalue spectra and dynamics of neural networks. Neural Comput 21(10):2931–2941. https://doi.org/10.1162/neco.2009.12-07-671
Zhou S, Masmanidis SC, Buonomano DV (2022) Encoding time in neural dynamic regimes with distinct computational tradeoffs. PLoS Comput Biol 18(3):1–29. https://doi.org/10.1371/journal.pcbi.1009271
Acknowledgements
The present work was supported by CONICET and UNQ. Authors acknowledge support from PICT 2020-01413.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
The code of this analysis is available at the following Github repository https://github.com/katejarne/Excitatory-inhibitory. Supplementary information in pdf file includes additional examples for all tasks and initial conditions. Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jarne, C., Caruso, M. Effect in the spectra of eigenvalues and dynamics of RNNs trained with excitatory–inhibitory constraint. Cogn Neurodyn (2023). https://doi.org/10.1007/s11571-023-09956-w
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11571-023-09956-w