Entropy conditions for Lr-convergence of empirical processes

  • Andrea Caponnetto
  • Ernesto De Vito
  • Massimiliano Pontil


The law of large numbers (LLN) over classes of functions is a classical topic of empirical processes theory. The properties characterizing classes of functions on which the LLN holds uniformly (i.e. Glivenko–Cantelli classes) have been widely studied in the literature. An elegant sufficient condition for such a property is finiteness of the Koltchinskii–Pollard entropy integral, and other conditions have been formulated in terms of suitable combinatorial complexities (e.g. the Vapnik–Chervonenkis dimension). In this paper, we endow the class of functions \(\mathcal F\) with a probability measure and consider the LLN relative to the associated Lr metric. This framework extends the case of uniform convergence over \(\mathcal F\), which is recovered when r goes to infinity. The main result is a Lr-LLN in terms of a suitable uniform entropy integral which generalizes the Koltchinskii–Pollard entropy integral.


Empirical processes Uniform entropy Rademacher averages Glivenko–Cantelli classes 

Mathematics Subject Classifications (2000)

Primary 60G15 60G51 


  1. 1.
    Alon, N., Ben-David, S., Cesa-Bianchi, N., Haussler, D.: Scale-sensitive dimensions, uniform convergence, and learnability. J. ACM 44(4), 616–631 (1997)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Conway, J.B.: A Course in Functional Analysis, 2nd edn. Springer-Verlag (1994)Google Scholar
  3. 3.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley (1991)Google Scholar
  4. 4.
    Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, New York (1996)MATHGoogle Scholar
  5. 5.
    Dudley, R.M.: A course on empirical processes. Lecture Notes in Math. 1097, 2–142 (1984)MathSciNetGoogle Scholar
  6. 6.
    Dudley, R.M.: Uniform Central Limit Theorems. Cambridge University Press, Cambridge, MA (1999)MATHGoogle Scholar
  7. 7.
    Kolmogorov, A.: Sulla determinazione empirica di una legge di distribuzione. Giorn. Ist. Ital. Attuari, A. IV(1), 4–11 (1933)Google Scholar
  8. 8.
    Koltchinskii, V.I.: On the central limit theorem for the empirical measures. Theory Probab. Math. Statist. 24, 71–82 (1981)MathSciNetGoogle Scholar
  9. 9.
    Pollard, D.: A central limit theorem for the empirical processes. J. Austral. Math. Soc. Ser. A 33, 235–248 (1982)MATHMathSciNetCrossRefGoogle Scholar
  10. 10.
    Mukherjee, S., Vapnik, V.: Multivariate density estimation: an SVM approach. CBCL Paper #170/AI Memo #1653, MIT, Cambridge, MA, April (1999)Google Scholar
  11. 11.
    Schwartz, L.: Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés (noyaux reproduisants). J. Anal. Math. 13, 115–256 (1964)MATHCrossRefGoogle Scholar
  12. 12.
    Shawe-Taylor, J., Dolia, A.: A new framework for probability density estimation (2006, preprint)Google Scholar
  13. 13.
    van der Vaart, A.W., Wellner, J.A.: Weak Convergence and Empirical Processes. Springer Series in Statistics (1996)Google Scholar
  14. 14.
    Vapnik, V.N.: Statistical Learning Theory. Wiley, New York (1998)MATHGoogle Scholar
  15. 15.
    Vapnik, V.N., Chervonenkis, A.: On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl. 16, 264–280 (1971)MATHCrossRefGoogle Scholar
  16. 16.
    Vapnik, V.N., Chervonenkis, A.: Theory of pattern recognition [in Russian]. Nauka, Moscow, 1974. German translation: Wapnik, W., Tscherwonenkis, A., Theorie der zeichenerkennung, Akademie-Verlag, Berlin (1979)Google Scholar
  17. 17.
    Vapnik, V.N., Chervonenkis, A.: Necessary and sufficient conditions for the uniform convergence of empirical means to their expectations. Theory Probab. Appl. 3, 532–553 (1981)Google Scholar

Copyright information

© Springer Science+Business Media, LLC. 2008

Authors and Affiliations

  • Andrea Caponnetto
    • 1
  • Ernesto De Vito
    • 2
  • Massimiliano Pontil
    • 3
  1. 1.Department of MathematicsCity University of Hong KongKowloon TongHong Kong
  2. 2.DSAUniversità di GenovaGenovaItaly
  3. 3.Department of Computer ScienceUniversity College LondonLondonUK

Personalised recommendations