Skip to main content

Lipschitz Widths

Constructive Approximation (2022)Cite this article

Abstract

This paper introduces a measure, called Lipschitz widths, of the optimal performance possible of certain nonlinear methods of approximation. Notably, those Lipschitz widths provide a theoretical benchmark for the approximation quality achieved via deep neural networks. The paper also discusses basic properties of the Lipschitz widths and their relation to entropy numbers and other well-known widths such as the Kolmogorov and the stable manifold widths. We show that Lipschitz widths with fixed Lipschitz constant and entropy numbers decay very similar, while when the Lipschitz constant grows with n, the Lipschitz width could be much smaller than the entropy numbers.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

  1. Benyamini, Y., Lindenstrauss, J.: Geometric Nonlinear Functional Analysis, Vol. 1, American Mathematical Society Colloquium Publications, 48, AMS, Providence, RI (2000)

  2. Borsuk, K.: Drei Sätze über die \(n\)-dimensionale euklidische Sphäre Fund. Math. 20, 177–191 (1933)

    MATH  Google Scholar 

  3. Carl, B.: Entropy numbers, s-numbers, and eigenvalue problems. J. Funct. Anal. 41, 290–306 (1981)

    MathSciNet  Article  Google Scholar 

  4. Carl, B., Stephani, I.: Entropy, compactness and the approximation of operators. Cambridge University Press, Cambridge (1990)

    Book  Google Scholar 

  5. Cohen, A., DeVore, R., Petrova, G., Wojtaszczyk, P.: Optimal stable nonlinear approximation, J. FoCM, to appear

  6. DeVore, R., Hanin, B., Petrova, G.: Neural network approximation. Acta Numerica 30, 327–444 (2021)

    MathSciNet  Article  Google Scholar 

  7. DeVore, R., Howard, R., Micchelli, C.: Optimal nonlinear approximation. Manuscripta Math. 63(4), 469–478 (1989)

    MathSciNet  Article  Google Scholar 

  8. DeVore, R., Kyriazis, G., Leviatan, D., Tichomirov, V.: Wavelet compression and nonlinear-widths. Adv. Comput. Math. 1(2), 197–214 (1993)

    MathSciNet  Article  Google Scholar 

  9. DeVore, R., Sharpley, R.: Besov spaces on domains in \(\mathbb{R}^d\). Trans. Am. Math. Soc. 335(2), 843–864 (1993)

    MATH  Google Scholar 

  10. Kolmogorov, A.N., Tihomirov, V.M.: \(\varepsilon \)-entropy and \(\varepsilon \)-capacity of sets in function spaces. (in Russian) Uspehi Mat. Nauk 14(2), 386 (1959)

    MathSciNet  Google Scholar 

  11. Kühn, T.: Entropy numbers of general diagonal operators. Rev. Mat. Compl. 18(2), 479–491 (2005)

    MathSciNet  MATH  Google Scholar 

  12. Lorentz, G.G., Golitschek, M., Makovoz, Y.: Constructive approximation, advanced problems Grundlehren der mathematischen Wissenschaften. Springer Verlag, Cham (1996)

    MATH  Google Scholar 

  13. Lu, J., Shen, Z., Yang, H., Zhang, S.: Deep network approximation for smooth functions. SIAM J. Math. Anal. 53(5), 5465–5506 (2020)

    MathSciNet  Article  Google Scholar 

  14. Pinkus, A.: \(n\)-Widths in Approximation Theory. Springer Verlag, Ergebnisse der Mathematik und ihrer Grenzgebiete (1985)

  15. Wojtaszczyk, P.: Banach spaces for analysts, Cambridge studies in advanced mathematics 25. Cambridge University Press, Cambridge (1991)

    Book  Google Scholar 

  16. Yarotsky, D.: Optimal approximation of continuous functions by very deep relu networks, Proceedings of the 31st Conference On Learning Theory, PMLR 75, 639–649 (2018)

Download references

Acknowledgements

We would like to thank the referee for the essential remarks and suggestions which helped improve the quality and readability of the paper. Among other things, the referee pointed out to us a simpler proof of Lemma 2.8 and a simpler example that is discussed in Remark 3.9.

Author information

Authors and Affiliations

  1. Department of Mathematics, Texas A &M University, College Station, TX, 77843, USA

    Guergana Petrova

  2. Institute of Mathematics Polish Academy of Sciences, ul. Śniadeckich 8, 00-656, Warszawa, Poland

    Przemysław Wojtaszczyk

Authors
  1. Guergana Petrova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Przemysław Wojtaszczyk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guergana Petrova.

Additional information

Communicated by Albert Cohen.

To Ron DeVore, with the utmost respect and admiration.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

G.P. was supported by the NSF Grant DMS 2134077, Tripods Grant CCF-1934904, and ONR Contract N00014-20-1-278. P. W. was supported by National Science Centre, Polish Grant UMO-2016/21/B/ST1/00241.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Petrova, G., Wojtaszczyk, P. Lipschitz Widths. Constr Approx (2022). https://doi.org/10.1007/s00365-022-09576-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00365-022-09576-3

Keywords

  • Widths
  • Entropy numbers
  • Neural networks

Mathematics Subject Classification

  • 41A46
  • 41A65
  • 82C32