Abstract
We had earlier constructed neural networks which are capable of providing optimal approximation rates for smooth target functions. The activation functions evaluated by the principal elements of these networks were infinitely many times differentiable. In this paper, we prove that the parameters of any network with these two properties must satisfy certain lower bounds. Our results can also be thought of as providing a rudimentary test for the hypothesis that the unknown target function belongs to a Sobolev class.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
C. K. Chui and X. Li, Approximation by ridge functions and neural networks with one hidden layer, J. Approx. Theory, Vol. 70 (1992), pp.131–141.
G. Cybenko, Approximation by superposition of sigmoidal functions, Mathematics of Control, Signal and Systems, Vol. 2 (1989), pp.303–314.
R. DeVore, R. Howard, and C. A. Micchelli, Optimal nonlinear approximation, Manuscripta Mathematica, Vol. 63 (1989), pp.469–478.
F. Girosi, M. Jones, and T. Poggio, Regularization theory and neural networks architectures, Neural Computation, Vol. 7 (1995), pp.219–269.
K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators, Neural Networks, Vol. 2 (1989), pp.359–366.
M. Leshno, V. Lin, A. Pinkus, and S. Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Networks, Vol. 6 (1993), pp.861–867.
H. N. Mhaskar, Neural networks for optimal approximation of smooth and analytic functions, Neural Computation, Vol.8 (1996), pp164–177.
H. N. Mhaskar and C. A. Micchelli, Approximation by superposition of a sigmoidal function and radial basis functions, Advances in Applied Mathematics, Vol. 13 (1992), pp.350–373.
H. N. Mhaskar and C. A. Micchelli, Dimension independent bounds on the degree of approximation by neural networks, IBM Journal of Research and Development, Vol. 38 (1994), pp.277–284.
H. N. Mhaskar and C. A. Micchelli, Degree of approximation by neural and translation networks with a single hidden layer, Advances in Applied Mathematics, Vol. 16 (1995), pp.151–183.
H. N. Mhaskar and C. A. Micchelli, How to choose an activation function, in NIPS*6 (J. D. Cowan, G. Tesauro, J. Alspector Eds.), Morgan Kaufmann, 1994, pp.319–326.
A. F. Timan, Theory of Approximation of Functions of a Real Variable, Macmillan Co., New York, 1963.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1997 Springer Science+Business Media New York
About this chapter
Cite this chapter
Mhaskar, H.N. (1997). On Smooth Activation Functions. In: Ellacott, S.W., Mason, J.C., Anderson, I.J. (eds) Mathematics of Neural Networks. Operations Research/Computer Science Interfaces Series, vol 8. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-6099-9_47
Download citation
DOI: https://doi.org/10.1007/978-1-4615-6099-9_47
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-7794-8
Online ISBN: 978-1-4615-6099-9
eBook Packages: Springer Book Archive