Abstract
Without doubt, the brain is the most complex adaptive system known to humanity, arguably also a complex system about which we know little. In both respects, the brain faces increasing competition from machine learning architectures.
We present an introduction to basic neural network and machine learning concepts, with a special focus on the connection to dynamical systems theory. Starting with point neurons and the XOR problem, the relation between the dynamics of recurrent networks and random matrix theory will be developed. The somewhat counter-intuitive notion of continuous numbers of network layers is shown next to lead to neural differential equations, respectively for information processing and error backpropagation. Approaches aimed at understanding learning processes in deep architectures make often use of the infinite-layer limit. As a result, machine learning can be described by Gaussian processes together with neural tangent kernels. Finally, the distinction between information processing and information routing will be discussed, with the latter being the task of the attention mechanism, the core component of transformer architectures.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
For the general theory of stochastic dynamical systems see Chap. 3.
- 3.
See Exercise (10.1).
- 4.
M. Minsky, S. Papert, “Perceptrons: An Introduction to Computational Geometry” (1969).
- 5.
Generic network theory is developed in Chap. 1.
- 6.
See Chap. 2.
- 7.
More about random variables in Chap. 5.
- 8.
- 9.
- 10.
- 11.
See exercise (10.7).
- 12.
References
Akjouj, I., et al. (2022). Complex systems in ecology: A guided tour with large Lotka-Volterra models and random matrices. arXiv:2212.06136.
Biehl, M. (2023). The shallow and the deep: A biased introduction to neural networks and old school machine learning. University of Groningen Press.
Chen, R. T., Rubanova, Y., Bettencourt, J., & Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31.
Dauphin, Y. N., Fan, A., Auli, M., & Grangier, D. (2017). Language modeling with gated convolutional networks. PMLR, 70, 933–941.
Gros, C. (2021). A Devil’s advocate view on ‘Self-Organized’ brain criticality. Journal of Physics: Complexity, 2, 2021.
Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Lindsay, G. W. (2020). Attention in psychology, neuroscience, and machine learning. Frontiers in Computational Neuroscience, 14, 29.
Sommers, H. J., Crisanti, A., Sompolinsky, H., & Stein, Y. (1988). Spectrum of large random asymmetric matrices. Physical Review Letters, 60, 1895.
Schubert, F., & Gros, C. (2021). Local homeostatic regulation of the spectral radius of echo-state networks. Frontiers in Computational Neuroscience, 15, 587721.
Sun, Y. et al. (2023). Retentive network: A successor to transformer for large language models. arXiv:2307.08621.
Vaswani, A. et al. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (vol. 30).
Williams, C. K., & Rasmussen, C. E. (2006). Gaussian processes for machine learning. MIT Press.
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019). A review of recurrent neural networks: LSTM cells and network architectures. Neural Computation, 31, 1235–1270.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Gros, C. (2024). Complexity of Machine Learning. In: Complex and Adaptive Dynamical Systems. Springer, Cham. https://doi.org/10.1007/978-3-031-55076-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-55076-8_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-55075-1
Online ISBN: 978-3-031-55076-8
eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)