Advertisement

Complexity of Shallow Networks Representing Functions with Large Variations

  • Věra Kůrková
  • Marcello Sanguineti
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8681)

Abstract

Model complexities of networks representing multivariable functions is studied in terms of variational norms tailored to types of network units. It is shown that the size of the variational norm reflects both the number of hidden units and sizes of output weights. Lower bounds on growth of variational norms with increasing input dimension d are derived for Gaussian units and perceptrons. It is proven that variation of the d-dimensional parity with respect to Gaussian Support Vector Machines grows exponentially with d and for large values of d, almost any randomly-chosen Boolean function has variation with respect to perceptrons depending on d exponentially.

Keywords

One-hidden-layer networks model complexity representations of multivariable functions perceptrons Gaussian SVMs 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bengio, Y.: Learning deep architectures for AI. Foundations and Trends in Machine Learning 2, 1–127 (2009)CrossRefzbMATHGoogle Scholar
  2. 2.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18, 1527–1554 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Chow, T.W.S., Cho, S.Y.: Neural Networks and Computing: Learning Algorithms and Applications. World Scientific (2007)Google Scholar
  4. 4.
    Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S.: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks 6, 861–867 (1993)CrossRefGoogle Scholar
  5. 5.
    Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143–195 (1999)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Park, J., Sandberg, I.: Approximation and radial-basis-function networks. Neural Computation 5, 305–316 (1993)CrossRefGoogle Scholar
  7. 7.
    Mhaskar, H.N.: Versatile Gaussian networks. In: Proc. of IEEE Workshop of Nonlinear Image Processing, pp. 70–73 (1995)Google Scholar
  8. 8.
    Kůrková, V.: Some comparisons of networks with radial and kernel units. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012, Part II. LNCS, vol. 7553, pp. 17–24. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Kainen, P.C., Kůrková, V., Sanguineti, M.: Dependence of computational models on input dimension: Tractability of approximation and optimization tasks. IEEE Transactions on Information Theory 58, 1203–1214 (2012)CrossRefGoogle Scholar
  10. 10.
    Maiorov, V.: On best approximation by ridge functions. J. of Approximation Theory 99, 68–94 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Maiorov, V., Pinkus, A.: Lower bounds for approximation by MLP neural networks. Neurocomputing 25, 81–91 (1999)CrossRefzbMATHGoogle Scholar
  12. 12.
    Bartlett, P.L.: The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans. on Information Theory 44, 525–536 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Bengio, Y., Delalleau, O., Roux, N.L.: The curse of highly variable functions for local kernel machines. In: Advances in Neural Information Processing Systems 18, pp. 107–114. MIT Press (2006)Google Scholar
  14. 14.
    Ito, Y.: Finite mapping by neural networks and truth functions. Mathematical Scientist 17, 69–77 (1992)zbMATHMathSciNetGoogle Scholar
  15. 15.
    Micchelli, C.A.: Interpolation of scattered data: Distance matrices and conditionally positive definite functions. Constructive Approximation 2, 11–22 (1986)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Barron, A.R.: Neural net approximation. In: Narendra, K. (ed.) Proc. 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press (1992)Google Scholar
  17. 17.
    Kůrková, V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing. The Curse of Dimensionality, pp. 261–270. Birkhäuser, Boston (1997)CrossRefGoogle Scholar
  18. 18.
    Kůrková, V., Sanguineti, M.: Comparison of worst-case errors in linear and neural network approximation. IEEE Transactions on Information Theory 48, 264–275 (2002)CrossRefzbMATHGoogle Scholar
  19. 19.
    Gnecco, G., Sanguineti, M.: On a variational norm tailored to variable-basis approximation schemes. IEEE Trans. on Information Theory 57, 549–558 (2011)CrossRefMathSciNetGoogle Scholar
  20. 20.
    Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Networks 11, 651–659 (1998)CrossRefGoogle Scholar
  21. 21.
    Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Networks 33, 160–167 (2012)CrossRefzbMATHGoogle Scholar
  22. 22.
    Schläfli, L.: Theorie der vielfachen Kontinuität. Zürcher & Furrer, Zürich (1901)CrossRefGoogle Scholar
  23. 23.
    Roychowdhury, V., Siu, K., Orlitsky, A.: Neural models and spectral methods. In: Roychowdhury, V., Siu, K., Orlitsky, A. (eds.) Theorertical Advances in Neural Computation and Learning, pp. 3–36. Kluwer Academic Publishers (1997)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Věra Kůrková
    • 1
  • Marcello Sanguineti
    • 2
  1. 1.Institute of Computer ScienceAcademy of Sciences of the Czech RepublicPragueCzech Republic
  2. 2.DIBRISUniversity of GenoaGenovaItaly

Personalised recommendations