Abstract
We propose a neural network architecture in infinite dimensional spaces for which we can show the universal approximation property. Indeed, we derive approximation results for continuous functions from a Fréchet space \(\mathfrak {X}\) into a Banach space \(\mathfrak {Y}\). The approximation results are generalising the well known universal approximation theorem for continuous functions from \(\mathbb {R}^{n}\) to \(\mathbb {R}\), where approximation is done with (multilayer) neural networks Cybenko (1989) Math. Cont. Signals Syst. 2, 303–314 and Hornik et al. (1989) Neural Netw., 2, 359–366 and Funahashi (1989) Neural Netw., 2, 183–192 and Leshno (1993) Neural Netw., 6, 861–867. Our infinite dimensional networks are constructed using activation functions being nonlinear operators and affine transforms. Several examples are given of such activation functions. We show furthermore that our neural networks on infinite dimensional spaces can be projected down to finite dimensional subspaces with any desirable accuracy, thus obtaining approximating networks that are easy to implement and allow for fast computation and fitting. The resulting neural network architecture is therefore applicable for prediction tasks based on functional data.
Similar content being viewed by others
Data Availability
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
Beck, C., Hutzenthaler, M., Jentzen, A., Kuckuck, B.: An overview on deep learning-based approximation methods for partial differential equations (2021)
Benth, F.E., Detering, N., Galimberti, L.: A functional neural network approach to the Cauchy problem (2022)
Benth, F.E., Detering, N., Galimberti, L.: Pricing options on flow forwards by neural networks in Hilbert space (2022)
Benth, F.E., Detering, N., Lavagnini, S.: Accuracy of deep learning in calibrating HJM forward curves. Digital Finance 3(3-4), 209–248 (2021)
Berner, J., Grohs, P., Kutyniok, G., Petersen, P.: The modern mathematics of deep learning (2021)
Bogachev, V.: Measure Theory. Number v. 1 in measure theory. Springer (2007)
Bogachev, V.: Measure Theory. Number v. 2 in measure theory. Springer (2007)
Brezis, H.: Functional analysis, sobolev spaces and partial differential equations. Universitext. Springer, New York (2010)
Buehler, H., Gonon, L., Teichmann, J., Wood, B.: Deep hedging. Quantitative Finance 19(8), 1271–1291 (2019)
Carmona, R.A., Tehranchi, M.R.: Interest rate models: an infinite dimensional stochastic analysis perspective. Springer (2006)
Chen, T., Chen, H.: Approximations of continuous functionals by neural networks with application to dynamic systems. IEEE Trans Neural Netw 4(6), 910–918 (1993)
Chen, T., Chen, H.: Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Trans Neural Netw 6(4), 911–917 (1995)
Cho, Y., Saul, L.K.: Kernel methods for deep learning. Advances in neural information processing systems:342–350 (2009)
Conway, J.B: A Course in Functional Analysis. Graduate Texts in Mathematics; 96, 2nd edn. Springer Science+Business Media, New York (2010)
Cuchiero, C., Larsson, M., Teichmann, J.: Deep neural networks, generic universal interpolation, and controlled odes. SIAM J Math Data Sci 2, 901–919 (2020)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Cont. Signals Syst. 2(4), 303–314 (1989)
Weinan, E.: A proposal on machine learning via dynamical systems. Commun Math Stat 5(1), 1–11 (2017)
Enflo, P.: A counterexample to the approximation problem in Banach spaces. Acta Math 130, 309–317 (1973)
Filipović, D: Consistency problems for heath-jarrow-morton interest rate models. Springer (2001)
Funahashi, K-I: On the approximate realization of continuous mappings by neural networks. Neural Netw. 2(3), 183–192 (1989)
Guss, W., Salakhutdinov, R.: On universal approximation by neural networks with uniform guarantees on approximation of infinite dimensional maps (2019)
Han, J., Jentzen, A., Weinan, E.: Solving high-dimensional partial differential equations using deep learning. Proc. National Academy Sci. 115(34), 8505–8510 (2018)
Hanin, B., Sellke, M.: Approximating continuous functions by relu nets of minimal width:10 (2017)
Hazan, T.: T Jaakola Steps toward deep kernel methods from infinite neural networks (2015)
Heil, C.: A basis theory primer: expanded edition. Applied and Numerical Harmonic Analysis birkhäuser Boston (2011)
Holden, H., Oksendal, B., Uboe, J., Zhang, T.: Stochastic partial differential equations: a modeling, white noise functional approach springer (2010)
Hornik, K.: Neural networks for functional approximation and system identification. Neural Comput. 9(1), 143–159 (1997)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)
Hutzenthaler, M., Jentzen, A., Kruse, T., Nguyen, T.A.: A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. SN Partial. Differ. Equ Appl. 1(2), 10 (2020)
Kidger, P., Lyons, T.: Universal approximation with deep narrow networks. In: Abernethy, J., Agarwal, S (eds.) Proceedings of thirty third conference on learning theory, vol. 125 of proceedings of machine learning research, pp. 2306–2327. PMLR, 09–12 Jul (2020)
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., Anandkumar, A.: Neural operator: learning maps between function spaces (2021)
Kratsios, A.: The universal approximation property. Annals Math. Artif. Intell. 89(5), 435–469 (2021)
Kratsios, A., Bilokopytov, I.: Non-euclidean universal approximation (2020)
Lanthaler, S., Mishra, S., Karniadakis, G.E.: Error estimates for DeepONets: a deep learning framework in infinite dimensions. Trans. Math. Appl. 6 (1), 03 (2022). tnac001
Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S.: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw. 6(6), 861–867 (1993)
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A.: Neural operator: graph kernel network for partial differential equations (2020)
Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.E.: Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Mach. Intell. 3(3), 218–229 (2021)
Lu, Z., Pu, H., Wang, F., Hu, Z., Wang, L.: The expressive power of neural networks: a view from the width 09 (2017)
Meise, R., Vogt, D.: Einführung in die Funktionalanalysis. Aufbaukurs Mathematik Vieweg (1992)
Mhaskar, H.N., Hahm, N.: Some new results on neural network approximation. Neural Netw. 6(8), 1069–1072 (1993)
Müller, D., Soto-Rey, I., Kramer, F.: An analysis on ensemble learning optimized medical image classification with deep convolutional neural networks (2022)
Narici, L.: Topological vector spaces. Monographs and textbooks in pure and applied mathematics; 95. M. Dekker, New York 1985 (1985)
Neal, R.M: Bayesian learning for neural networks lecture notes in statistics:118. Springer science+business media, New York (1996)
Pinkus, A.: Approximation theory of the mlp model in neural networks. Acta Numer 8, 143–195 (1999)
Ramsey, J.O., Silverman, B.W.: Functional Data Analysis, 2nd edn. Springer Science+Business Media, New York (2005)
Sandberg, I.: Approximation theorems for discrete-time systems. IEEE Trans. Circuits Syst. 38(5), 564–566 (1991)
Schaefer, H.: Topological vector spaces. Elements of mathematics / N. Bourbaki Springer (1971)
Schwartz, L.: Théorie des distributions. Number v. 1-2 in actualités scientifiques et industrielles Hermann (1957)
Tian, T.S.: Functional data analysis in brain imaging studies. Frontiers Psychol. 1, 35–35,10 (2010)
Triebel, H.: A note on wavelet bases in function spaces. Banach Center Publ. 64(1), 193–206 (2004)
Williams, C.K.I: Computing with infinite networks. Adv. Neural Inf. Process. Syst., pp. 295–301 (1997)
Yu, P., Yan, X.: Stock price prediction based on deep neural networks. Neural Comput. Appl. 32(6), 1609–1628 (2020)
Acknowledgements
We are grateful to an anonymous referee for the constructive critics leading to a significant improvement of this paper. Fred Espen Benth acknowledges support from SPATUS, a Thematic Research Group funded by UiO:Energy. Luca Galimberti has been supported in part by the grant Waves and Nonlinear Phenomena (WaNP) from the Research Council of Norway.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Benth, F.E., Detering, N. & Galimberti, L. Neural networks in Fréchet spaces. Ann Math Artif Intell 91, 75–103 (2023). https://doi.org/10.1007/s10472-022-09824-z
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-022-09824-z