Low-Dimensional Dynamics of Encoding and Learning in Recurrent Neural Networks

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12109)


In this paper, we use dimensionality reduction techniques to study how a recurrent neural network (RNN) processes and encodes information in the context of a classification task, and we explain our findings using tools from dynamical systems theory. We observe that internal representations develop a task-relevant structure as soon as significant information is provided as input and this structure remains for some time even if we let the dynamics drift. However, the structure is only interpretable by the final classifying layer at the fixed time step for which the network was trained. We measure that throughout the training, the recurrent weights matrix is modified so that the resulting dynamical system associated with the network’s neural activations evolves into a non-trivial attractor, reminiscent of neural oscillations in the brain. Our findings suggest that RNNs change their internal dynamics throughout training so that information is stored in low-dimensional cycles, rather than in high-dimensional clusters.


Recurrent neural networks Internal representations geometry Dynamical systems 



We would like to thank Aude Forcione-Lambert and Giancarlo Kerg for useful discussions. This work was partially funded by: CRM-ISM scholarships [S.H., V.G.]; IVADO (l’institut de valorisation des données), NIH grant R01GM135929 [G.W.]; NSERC Discovery Grant (RGPIN-2018-04821), FRQNT Young Investigator Startup Program (2019-NC-253251), and FRQS Research Scholar Award, Junior 1 (LAJGU0401-253188) [G.L.] The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.


  1. 1.
    Bengio, Y., Mikolov, T., Pascanu, R.: On the difficulty of training Recurrent Neural Networks. arXiv e-prints, November 2012Google Scholar
  2. 2.
    Cohen, U., Chung, S., Lee, D.D., Sompolinsky, H.: Separability and geometry of object manifolds in deep neural networks. bioRxiv (2019)Google Scholar
  3. 3.
    Crisanti, A., Sompolinsky, H.: Path integral approach to random neural networks. Phys. Rev. E 98(6), 062120 (2018)CrossRefGoogle Scholar
  4. 4.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  5. 5.
    Marquez, B.A., Larger, L., Jacquot, M., Chembo, Y.K., Brunner, D.: Dynamical complexity and computation in recurrent neural networks beyond their fixed point. Sci. Rep. 8(1), 3319 (2018)CrossRefGoogle Scholar
  6. 6.
    McInnes, L., Healy, J.: UMAP: Uniform manifold approximation and projection for dimension reduction. ArXiv 1802.03426 (2018)Google Scholar
  7. 7.
    Pina, J., Bodner, M., Ermentrout, B.: Oscillations in working memory and neural binding: a mechanism for multiple memories and their interactions. PLoS Comput. Biol. 14(11), 1–30 (2018)CrossRefGoogle Scholar
  8. 8.
    Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., Ganguli, S.: Exponential expressivity in deep neural networks through transient chaos. arXiv e-prints (2016)Google Scholar
  9. 9.
    Sompolinsky, H., Crisanti, A., Sommers, H.J.: Chaos in random neural networks. Phys. Rev. Lett. 61, 259–262 (1988)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Sussillo, D., Barak, O.: Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25, 626–649 (2012) Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Mathematics and StatisticsUniversité de MontréalMontrealCanada
  2. 2.Mila - Quebec Artificial Intelligence InstituteMontrealCanada

Personalised recommendations