Summary
An estimate m n of a regression function m(x)=E{Y|X=x} is weakly (strongly) consistent in L 1 if ∝¦m n (x)-m(x)¦μ(dx) converges to 0 in probability (w.p. 1) as the sample size grows large (μ is the probability measure of X).
We show that the well-known kernel estimate (Nadaraya, Watson) and several recursive modifications of it are weakly (strongly) consistent in L 1 under no conditions on (X, Y) other than the boundedness of Y and the absolute continuity of μ. No continuity restrictions are put on the density corresponding to μ. We further notice that several kernel-type discrimination rules are weakly (strongly) Bayes risk consistent whenever X has a density.
Article PDF
Similar content being viewed by others
References
Ahmad, I.A., Lin, P.: Nonparametric sequential estimation of a multiple regression function. Bull. Math. Statist. 17, 63–75 (1976)
Aizerman, M.A., Braverman, E.M., Rozonoer, L.I.: Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control 25, 917–936 (1964a)
Aizerman, M.A., Braverman, E.M., Rozonoer, L.I.: The probability problem of pattern recognition learing and the method of potential functions. Automat. Remote Control 25, 1307–1323 (1964b)
Bashkirov, O.A., Braverman, E.M., Muchnik, I.E.: Potential function algorithms for pattern recognition learning machines. Automat. Remote Control 25, 692–695 (1964)
Bennett, G.: Probability inequalities for the sum of independent random variables. J. Amer. Statist. Assoc. 57, 33–45 (1962)
Cacoullos, T.: Estimation of a multivariate density. Ann. Inst. Statist. Math. 18, 179–190 (1965)
Devroye, L.P., Wagner, T.J.: Nonparametric discrimination and density estimation. Electronics Research Center, Univ. of Texas. Technical Report 183. 1976.
Devroye, L., Wagner, T.J.: Distribution free consistency results in nonparametric discrimination and regression function estimation. [To appear in Ann. Statist. 1979]
Fix, E., Hodges, J.L.: Discriminatory analysis. Nonparametric discrimination: consistency properties. USAF School of Aviation Medicine, Randolph Field, Texas. Report 4. Project No. 21-49-004. 1951
Glick, N.: Sample-based classification procedures derived from density estimators. J. Amer. Statist. Assoc. 67, 116–122 (1972)
Glick, N.: Sample-based classification procedures related to empiric distributions. IEEE Trans. Information Theory IT-22, 454–461 (1976)
Greblicki, W.: Asymptotically optimal probabilistic algorithms for pattern recognition and identification. Monografie No. 3. Prace Naukowe Instytutu Cybernetyki Technicznej Politechniki Wroclawsjiej Nr. 18. Wroclaw Poland 1974.
Greblicki, W.: Pattern recognition procedures with nonparametric density estimates. Submitted to the IEE Trans. Systems, Man and Cybernetics (1977)
Greblicki, W.: Asymptotically optimal pattern recognition procedures with density estimates. To appear in the IEEE Trans. Information Theory (1978)
Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Amer. Statist. Assoc. 58, 13–30 (1963)
Loève, M.: Probability theory. Princeton, New Jersey: Van Nostrand 1963
Meisel, W.: Potential functions in mathematical pattern recognition. IEEE Trans. Computers 18, 911–918 (1969)
Nadaraya, E.A.: On estimating regression. Theor. Probab. Appl. 9, 141–142 (1964)
Nadaraya, E.A.: Remarks on nonparametric estimates for density functions and regression curves. Theor. Probab. Appl. 15, 134–137 (1970)
Noda, K.: Estimation of a regression function by the Parzen kernel-type density estimators. Ann. Inst. Statist. Math. 28, 221–234 (1976)
Parzen, E.: On the estimation of a probability density function and the mode. Ann. Math. Statist. 33, 1065–1076 (1962)
Rejtö, L., Révész, P.: Density estimation and pattern classification. Problems of Control and Information Theory 2, 67–80 (1973)
Révész, P.: Robbins-Monroe procedure in a Hilbert space and its application in the theory of learning processes. I. Studia Sci. Math. Hungar. 8, 391–398 (1973)
Révész, P.: How to apply the method of stochastic approximation in the nonparametric estimation of a regression function. Math. Operationsforsch. Statist. Ser. Statist. 8, 119–126 (1977)
Rosenblatt, M.: Remarks on some nonparametric estimates of a density function. Ann. Math. Statist. 27, 832–837 (1957)
Rosenblatt, M.: Conditional probability density and regression estimators. Multivariate Analysis II. pp. 25–31. P.R. Krishnaiah Ed. New York: Academic Press 1969
Schuster, E.F.: Joint asymptotic distribution of the estimated regression function at a finite number of distinct points. Ann. Math. Statist. 43, 84–88 (1972)
Sebestyen, G.: Decision making processes in pattern recognition. New York: Macmillan 1962
Stein, E.M.: Singular integrals and differentiability properties of functions. Princeton, New Jersey: Princeton Univ. Press 1970
Stone, C.J.: Consistent nonparametric regression. Annals of Statistics 5, 595–645 (1977)
Van Ryzin, J.: Bayes risk consistency of classification procedures using density estimation. Sankhyā Ser. A28, 161–170 (1966)
Van Ryzin, J.: A stochastic a posteriori updating algorithm for pattern recognition. J. Math. Anal. Appl. 20, 359–379 (1967)
Watson, G.S.: Smooth regression analysis. Sankhyā Ser. A 26, 359–372 (1964)
Wolverton, C.T., Wagner, T.J.: Asymptotically optimal discriminant functions for pattern classification. IEEE Trans. Information Theory IT-15, 258–265 (1969)
Author information
Authors and Affiliations
Additional information
Research of both authors was sponsored by AFOSR Grant 77-3385
Rights and permissions
About this article
Cite this article
Devroye, L.P., Wagner, T.J. On the L 1 convergence of kernel estimators of regression functions with applications in discrimination. Z. Wahrscheinlichkeitstheorie verw Gebiete 51, 15–25 (1980). https://doi.org/10.1007/BF00533813
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF00533813