Low-Dimensional Dynamics of Encoding and Learning in Recurrent Neural Networks
- 827 Downloads
In this paper, we use dimensionality reduction techniques to study how a recurrent neural network (RNN) processes and encodes information in the context of a classification task, and we explain our findings using tools from dynamical systems theory. We observe that internal representations develop a task-relevant structure as soon as significant information is provided as input and this structure remains for some time even if we let the dynamics drift. However, the structure is only interpretable by the final classifying layer at the fixed time step for which the network was trained. We measure that throughout the training, the recurrent weights matrix is modified so that the resulting dynamical system associated with the network’s neural activations evolves into a non-trivial attractor, reminiscent of neural oscillations in the brain. Our findings suggest that RNNs change their internal dynamics throughout training so that information is stored in low-dimensional cycles, rather than in high-dimensional clusters.
KeywordsRecurrent neural networks Internal representations geometry Dynamical systems
We would like to thank Aude Forcione-Lambert and Giancarlo Kerg for useful discussions. This work was partially funded by: CRM-ISM scholarships [S.H., V.G.]; IVADO (l’institut de valorisation des données), NIH grant R01GM135929 [G.W.]; NSERC Discovery Grant (RGPIN-2018-04821), FRQNT Young Investigator Startup Program (2019-NC-253251), and FRQS Research Scholar Award, Junior 1 (LAJGU0401-253188) [G.L.] The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
- 1.Bengio, Y., Mikolov, T., Pascanu, R.: On the difficulty of training Recurrent Neural Networks. arXiv e-prints, November 2012Google Scholar
- 2.Cohen, U., Chung, S., Lee, D.D., Sompolinsky, H.: Separability and geometry of object manifolds in deep neural networks. bioRxiv (2019)Google Scholar
- 6.McInnes, L., Healy, J.: UMAP: Uniform manifold approximation and projection for dimension reduction. ArXiv 1802.03426 (2018)Google Scholar
- 8.Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., Ganguli, S.: Exponential expressivity in deep neural networks through transient chaos. arXiv e-prints (2016)Google Scholar
- 10.Sussillo, D., Barak, O.: Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25, 626–649 (2012) Google Scholar