Abstract
We investigate the problem of successfully learning from just a few examples of data points in a binary classification problem, and present a brief overview of some recent results on the role of nonlinear feature maps in this challenging task. Our main conclusion is that successful learning and generalisation may be expected to occur with high probability, despite the small training sample, when the nonlinear feature map induces certain fundamental geometric properties in the mapped data.
Keywords
- Few-shot learning
- High dimensional data
- Nonlinear feature maps
The authors are grateful for financial support by the UKRI and EPSRC (UKRI Turing AI Fellowship ARaISE EP/V025295/1). I.Y.T. is also grateful for support from the UKRI Trustworthy Autonomous Systems Node in Verifiability EP/V026801/1.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bartlett, P., Long, P., Lugosi, G., Tsigler, A.: Benign overfitting in linear regression. Proc. Natl. Acad. Sci. 117(48), 30063–30070 (2020)
Bastounis, A., et al.: The crisis of empirical risk in verifiable, robust and accurate learning (2023, in preparation)
Bastounis, A., Hansen, A.C., Vlačić, V.: The mathematics of adversarial attacks in AI - why deep learning is unstable despite the existence of stable neural networks (2021). https://doi.org/10.48550/ARXIV.2109.06098. https://arxiv.org/abs/2109.06098
Gorban, A.N., Tyukin, I.Y.: Stochastic separation theorems. Neural Netw. 94, 255–259 (2017)
Gorban, A., Tyukin, I., Prokhorov, D., Sofeikov, K.: Approximation with random bases: pro et contra. Inf. Sci. 364–365, 129–145 (2016)
Kainen, P.C., Kůrková, V.: Quasiorthogonal dimension. In: Kosheleva, O., Shary, S.P., Xiang, G., Zapatrin, R. (eds.) Beyond Traditional Probabilistic Data Processing Techniques: Interval, Fuzzy etc. Methods and Their Applications. SCI, vol. 835, pp. 615–629. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-31041-7_35
Kainen, P.C., Kurková, V.: Quasiorthogonal dimension of Euclidean spaces. Appl. Math. Lett. 6(3), 7–10 (1993)
Ledoux, M.: The Concentration of Measure Phenomenon. No. 89, American Mathematical Soc. (2001)
Mallinar, N., Simon, J.B., Abedsoltan, A., Pandit, P., Belkin, M., Nakkiran, P.: Benign, tempered, or catastrophic: a taxonomy of overfitting. arXiv preprint arXiv:2207.06569 (2022)
Smola, A., Gretton, A., Song, L., Schölkopf, B.: A Hilbert space embedding for distributions. In: Hutter, M., Servedio, R.A., Takimoto, E. (eds.) ALT 2007. LNCS (LNAI), vol. 4754, pp. 13–31. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-75225-7_5
Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems, pp. 4077–4087 (2017)
Sutton, O.J., Gorban, A.N., Tyukin, I.Y.: Towards a mathematical understanding of learning from few examples with nonlinear feature maps (2022). https://doi.org/10.48550/ARXIV.2211.03607. https://arxiv.org/abs/2211.03607
Tyukin, I.Y., Gorban, A.N., Alkhudaydi, M.H., Zhou, Q.: Demystification of few-shot and one-shot learning. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE (2021)
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems, pp. 3630–3638 (2016)
Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: a survey on few-shot learning. ACM Comput. Surv. 53(3) (2020). https://doi.org/10.1145/3386252
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sutton, O.J., Gorban, A.N., Tyukin, I.Y. (2023). A Geometric View on the Role of Nonlinear Feature Maps in Few-Shot Learning. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2023. Lecture Notes in Computer Science, vol 14071. Springer, Cham. https://doi.org/10.1007/978-3-031-38271-0_56
Download citation
DOI: https://doi.org/10.1007/978-3-031-38271-0_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38270-3
Online ISBN: 978-3-031-38271-0
eBook Packages: Computer ScienceComputer Science (R0)