Automation and Remote Control

, Volume 80, Issue 11, pp 1949–1975 | Cite as

Complete Statistical Theory of Learning

  • V. N. VapnikEmail author
Topical Issue


Existing mathematical model of learning requires using training data find in a given subset of admissible function the function that minimizes the expected loss. In the paper this setting is called Second selection problem. Mathematical model of learning in this paper along with Second selection problem requires to solve the so-called First selection problem where using training data one first selects from wide set of function in Hilbert space an admissible subset of functions that include the desired function and second selects in this admissible subset a good approximation to the desired function. Existence of two selection problems reflects fundamental property of Hilbert space, existence of two different concepts of convergence of functions: weak convergence (that leads to solution of the First selection problem) and strong convergence (that leads to solution of the Second selection problem). In the paper we describe simultaneous solution of both selection problems for functions that belong to Reproducing Kernel Hilbert space. The solution is obtained in closed form.


statistical learning theory first selection problem second selection problem reproducing kernel Hilbert space training data 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vapnik, V.N. and Chervonenkis, A.J., Necessary and Sufficient Convergence of Relative Frequencies of events to Their Probabilities, Theor. Prob. App., 1971, vol. 16, no. 2, pp. 264–280.CrossRefGoogle Scholar
  2. 2.
    Vapnik, V.N. and Chervonenkis, A.J., Teoriya raspoznavaniya obrazov (Theory of Pattern Recognition), Moscow: Nauka, 1974.Google Scholar
  3. 3.
    Vapnik, V., The Nature of Statistical Learning Theory, New York: Springer, 1995.CrossRefGoogle Scholar
  4. 4.
    Vapnik, V.N., Statistical Learning Theory, New York: Wiley, 1998.zbMATHGoogle Scholar
  5. 5.
    Vapnik, V.N., Estimation of Dependencies Based on Empirical Data, Moscow: Nauka, 1979.Google Scholar
  6. 6.
    Devroy, L., Geofry, L., and Lugosi, G., A Probabilistic Theory of Pattern Recognition, New York: Springer, 1996.CrossRefGoogle Scholar
  7. 7.
    Tikhonov, A.N. and Arsenin, V.Ya., Solutions of Ill-Posed Problems, Washington: Winston, 1977.zbMATHGoogle Scholar
  8. 8.
    Vapnik, V. and Izmailov, R., Rethinking Statistical Learning Theory: Learning Using Statistical Invariants, Machine Learning, 2018, vol. 108, pp. 381–423.MathSciNetCrossRefGoogle Scholar

Copyright information

© Pleiades Publishing, Ltd. 2019

Authors and Affiliations

  1. 1.Columbia UniversityNew YorkUSA

Personalised recommendations