Skip to main content
SpringerLink
Log in

Critical points for least-squares problems involving certain analytic functions, with applications to sigmoidal nets

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

This paper deals with nonlinear least-squares problems involving the fitting to data of parameterized analytic functions. For generic regression data, a general result establishes the countability, and under stronger assumptions finiteness, of the set of functions giving rise to critical points of the quadratic loss function. In the special case of what are usually called “single-hidden layer neural networks”, which are built upon the standard sigmoidal activation tanh(x) (or equivalently (1 +e −x)−1), a rough upper bound for this cardinality is provided as well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. F. Albertini, E.D. Sontag and V. Maillot, Uniqueness of weights for neural networks, in:Artificial Neural Networks with Applications in Speech and Vision, ed. R. Mammone, Chapman and Hall, London, 1993, pp. 115–125.

    Google Scholar 

  2. E. Bierstone and P.D. Milman, Semianalytic and subanalytic sets, Inst. Hautes Études Sci. Publ. Math. 67, 1988, 5–42.

    Google Scholar 

  3. E.K. Blum, Approximation of Boolean functions by sigmoidal networks: Part I: XOR and other two-variable functions, Neural Computation 1, 1989, 532–540.

    Google Scholar 

  4. M. Brady, R. Raghavan and J. Slawny, Backpropagation fails to separate where perceptrons succeed, IEEE Trans. Circuits and Systems 36, 1989, 665–674.

    Article  Google Scholar 

  5. J.B. Conway,Regular Algebra and Finite Machines, Chapman and Hall, London, 1971.

    Google Scholar 

  6. A. Gabrielov, Projections of semi-analytic sets, Functional Anal. Appl. 2, 1968, 282–291.

    Article  Google Scholar 

  7. S.A. Goldman and M.J. Kearns, On the complexity of teaching,Proc. 4th ACM Workshop on Computational Learning Theory, July 1991, pp. 303–314.

  8. M. Gori and A. Tesi, On the problem of local minima in back-propagation, Tech. Report RT-DSI 6/90, Univ. di Firenze, April 1990.

  9. A.G. Khovanskii,Fewnomials, American Mathematical Society, Providence, RI, 1991.

    Google Scholar 

  10. J. Knight, A. Pillay and C. Steinhorn, Definable sets in ordered structures, II, Trans. Amer. Math. Soc. 295, 1986, 593–605.

    Google Scholar 

  11. A. Macintyre and E.D. Sontag, Finiteness results for sigmoidal “neural” networks, in:Proc. 25th Annual Symp. Theory Computing, San Diego, May 1993, pp. 325–334.

  12. J.W. Milnor,Morse Theory, Princeton University Press, 1963.

  13. R. Palais and C-l. Terng,Critical Point Theory and Submanifold Geometry, Springer, Berlin, New York, 1988.

    Google Scholar 

  14. T. Poston, C-N Lee, Y-J Choie and Y. Kwon, Local minima and backpropagation, in:Int. Joint Conf. Neural Networks, Seattle, IEEE Press, 1991, pp. 173–176.

  15. E.D. Sontag,Mathematical Control Theory: Deterministic Finite Dimensional Systems, Springer, New York, 1990.

    Google Scholar 

  16. E.D. Sontag, Feedforward nets for interpolation and classification, J. Comp. Syst. Sci. 45, 1992, 20–48.

    Article  Google Scholar 

  17. E.D. Sontag and H.J. Sussmann, Backpropagation can give rise to spurious local minima even for networks without hidden layers, Complex Systems 3, 1989, 91–106.

    Google Scholar 

  18. H.J. Sussmann, Real analytic desingularization and subanalytic sets: An elementary approach, Trans. Amer. Math. Soc. 317, 1990, 417–461.

    Google Scholar 

  19. H.J. Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, Neural Networks, 5, 1992, 589–593.

    Google Scholar 

  20. L. van den Dries, A generalization of the Tarski-Seidenberg theorem, and some nondefinability results, Bull. AMS 15, 1986, 189–193.

    Google Scholar 

  21. L. van den Dries, Tame topology and 0-minimal structures, preprint, University of Illinois, Urbana, 1991–2.

    Google Scholar 

  22. L. van den Dries and C. Miller, On the real exponential field with restricted analytic functions, Israel J. Math., 85 1994, 19–56.

    Google Scholar 

  23. L. van den Dries, A. Macintyre and D. Marker, The elementary theory of restricted analytic fields with exponentiation, Annals of Math. 140, 1994, 183–205.

    Google Scholar 

  24. R.C. Williamson and U. Helmke, Approximation theoretic results for neural networks, in:Proceedings of the Australian Conference on Neural Networks, 1992, pp. 217–222. (Also Existence and uniqueness results for neural network approximations, IEEE Transactions on Neural Networks, 6, 1995, 2–13.)

    Google Scholar 

  25. A.J. Wilkie, Some model completeness results for expansions of the ordered field of reals by Pfaffian functions, preprint, Oxford, 1991, submitted.

  26. A.J. Wilkie, Smooth 0-minimal theories and the model completeness of the real exponential field, preprint, Oxford, 1991, submitted.

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Rutgers University, 089003, New Brunswick, NJ, USA

    Eduardo D. Stonag

Authors
  1. Eduardo D. Stonag
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Supported in part by US Air Force Grant AFOSR-94-0293.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stonag, E.D. Critical points for least-squares problems involving certain analytic functions, with applications to sigmoidal nets. Adv Comput Math 5, 245–268 (1996). https://doi.org/10.1007/BF02124746

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02124746

Keywords

Search

Navigation