Abstract
This paper deals with sparse approximations by means of convex combinations of elements from a predetermined “basis” subsetS of a function space. Specifically, the focus is on therate at which the lowest achievable error can be reduced as larger subsets ofS are allowed when constructing an approximant. The new results extend those given for Hilbert spaces by Jones and Barron, including, in particular, a computationally attractive incremental approximation scheme. Bounds are derived for broad classes of Banach spaces; in particular, forL p spaces with 1<p<∞, theO (n −1/2) bounds of Barron and Jones are recovered whenp=2.
One motivation for the questions studied here arises from the area of “artificial neural networks,” where the problem can be stated in terms of the growth in the number of “neurons” (the elements ofS) needed in order to achieve a desired error rate. The focus on non-Hilbert spaces is due to the desire to understand approximation in the more “robust” (resistant to exemplar noise)L p, 1 ≤p<2, norms.
The techniques used borrow from results regarding moduli of smoothness in functional analysis as well as from the theory of stochastic processes on function spaces.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
S. Banach, S. Saks (1930):Sur la convergence forte dans les champs L P. Studia Math.,2:51–57.
A. R. Barron (1991):Approximation and estimation bounds for artificial neural networks. In: Proc. Fourth Annual Workshop on Computational Learning Theory. Morgan Kaufmann, pp. 243–249.
A. R. Barron (1992):Neural net approximation. In: Proc. of the Seventh Yale Workshop on Adaptive and Learning Systems. pp. 69–72.
C. Bessaga, A. Pelczynski (1958):A generalization of results of R. C. James concerning absolute bases in Banach spaces. Studia Math.,17:151–164.
J. A. Clarkson (1936):Uniformly convex spaces. Trans. Amer. Math. Soc.,40:396–414.
C. Darken, M. Donahue, L. Gurvits, E. Sontag (1993):Rate of approximation results motivated by robust neural network learning. In: Proc. of the Sixth Annual ACM Conference on Computational Learning Theory. New York: The Association for Computing Machinery. pp. 303–309.
R. Deville, G. Godefroy, V. Zizler (1993): Smoothness and Renormings in Banach Spaces. New York: Wiley.
J. Dieudonné (1960): Foundations of Modern Analysis. New York: Academic Press.
T. Figiel, G. Pisier (1974):Séries aléatoires dans les espaces uniformément convexes ou uniformément lisses. C. R. Acad. Sci. Paris,279:611–614.
W. T. Gowers (Preprint):A Banach space not containing c 0, l1, or a reflexive subspace.
U. Haagerup (1982):The best constants in the Khintchine inequality. Studia Math.,70:231–283.
O. Hanner (1956):On the uniform convexity of L p and ℓp. Arkiv. Matematik,3:239–244.
S. J. Hanson, D. J. Burr (1988):Minkowski-r back-propagation: learning in connectionist models with non-Euclidean error signals. In: Neural Information Processing Systems. New York: American Institute of Physics, p. 348.
R. C. James (1978):Nonreflexive spaces of type 2. Israel J. Math.,30:1–13.
L. K. Jones (1992):A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist.,20:608–613.
S. Kakutani (1938):Weak convergence in uniformly convex spaces. Tôhoku Math. J.,45:188–193.
J. Khintchine (1923):Über die diadischen Brüche. Math. Z.,18:109–116.
M. Ledoux, M. Talagrand (1991): Probability in Banach Space. Berlin: Springer-Verlag
M. Leshno, V. Lin, A. Pinkus, S. Schocken (1992):Multilayer feedforward networks with a non-polynomial activation function can approximate any function. Preprint. Hebrew University.
J. Lindenstrauss (1963):On the modulus of smoothness and divergent series in Banach spaces. Michigan Math. J.,10:241–252.
J. Lindenstrauss, L. Tzafriri (1979): Classical Banach Spaces II: Function Spaces. Berlin: Springer-Verlag.
M. J. D. Powell (1981): Approximation Theory and Methods. Cambridge: Cambridge University Press.
W. J. Rey (1983): Introduction to Robust and Quasi-Robust Statistical Methods. Berlin: Springer-Verlag.
H. Rosenthal (1974):A characterization of Banach spaces containing l l. Proc. Nat. Acad. Sci. (USA),71:2411–2413.
H. Rosenthal (1994):A subsequence principle characterizing Banach spaces containing c 0. Bull. Amer. Math. Soc.,30:227–233.
E. D. Sontag (1992):Feedback stabilization using two-hidden-layer nets. IEEE Trans. Neural Networks,3:981–990.
K. R. Stromberg (1981): An Introduction to Classical Real Analysis. New York: Wadsworth.
J. Y. T. Woo (1973):On modular sequence spaces. Studia Math.,48:271–289.
Author information
Authors and Affiliations
Additional information
Communicated by Vladimir N. Temlyakov.
Rights and permissions
About this article
Cite this article
Donahue, M.J., Darken, C., Gurvits, L. et al. Rates of convex approximation in non-hilbert spaces. Constr. Approx 13, 187–220 (1997). https://doi.org/10.1007/BF02678464
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02678464