Numerische Mathematik

, Volume 66, Issue 1, pp 41–66 | Cite as

Asymptotic optimality of generalized cross-validation for choosing the regularization parameter

  • Mark A. Lukas
Article
Received:
Revised:

Summary

Letf n λ be the regularized solution of a general, linear operator equation,Kf 0 =g, from discrete, noisy datay i =g(x) +ɛ i ,i=1,...,n, whereɛ i are uncorrelated random errors. We consider the prominent method of generalized cross-validation (GCV) for choosing the crucial regularization parameter λ. The practical GCV estimate\(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\lambda } _V \) and its “expected” counterpart λ V are defined as the minimizers of the GCV functionV(λ) andEV(λ), respectively, whereE denotes expectation. We investigate the asymptotic performance of λ V with respect to each of the following loss functions: the risk, anL2 norm on the output errorKf n λ−g, and a whole class of stronger norms on the input errorf n λ−f0. In the special cases of data smoothing and Fourier differentiation, it is known that asn→∞, λ V is asymptotically optimal (ao) with respect to the risk criterion. We show this to be true in general, and also extend it to theL2 norm criterion. The asymptotic optimality is independent of the error variance, the ill-posedness of the problem and the smoothness index of the solutionf0. For the input error criterion, it is shown that λ V is weakly ao for a certain class off0 if the smoothness off0 relative to the regularization space is not too high, but otherwise λ V is sub-optimal. This result is illustrated in the case of numerical differentiation.

Mathematics Subject Classifications (1991)

65J10 62G05 47A50 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cox, D.D. (1983): Asymptotics for M-type smoothing splines. Ann. Stat.11, 530–551Google Scholar
  2. 2.
    Cox, D.D. (1984): Multivariate smoothing spline functions. SIAM J. Numer. Anal.21, 789–813Google Scholar
  3. 3.
    Cox, D.D. (1988): Approximation of method of regularization estimators. Ann. Stat.16, 694–712Google Scholar
  4. 4.
    Craven, P., Wahba, G. (1979): Smoothing noisy data with spline functions. Numer. Math.31, 377–403Google Scholar
  5. 5.
    Davies, A.R., Anderssen, R.S. (1986): Improved estimates of statistical regularization parameters in Fourier differentiation and smoothing. Numer. Math.48, 671–697Google Scholar
  6. 6.
    Davies, A.R., Anderssen, R.S. (1986): Optimization in the regularization of ill-posed problems. J. Austral. Math. Soc. Ser. B28, 114–133Google Scholar
  7. 7.
    Golub, G.H., Heath, M., Wahba, G. (1979): Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics21, 215–223Google Scholar
  8. 8.
    Kimeldorf, G.S., Wahba, G. (1971): Some results on Tchebycheffian spline functions. J. Math. Anal. Appl.33, 82–95Google Scholar
  9. 9.
    Li, K.C. (1986): Asymptotic optimality ofC L and generalized cross-validation in ridge regression with application to spline smoothing. Ann. Stat.14, 1101–1112Google Scholar
  10. 10.
    Lukas, M.A. (1988): Convergence rates for regularized solutions. Math. Comp.51, 107–131Google Scholar
  11. 11.
    Lukas, M.A. (1988): Assessing regularized solutions. J. Austral. Math. Soc. Ser. B30, 24–42Google Scholar
  12. 12.
    Lukas, M.A. (1991): Asymptotic behaviour of the discrepancy principle and generalized maximum likelihood for choosing the regularization parameter. Report No. 4, Centre for Mathematics and its Applications, Australian National UniversityGoogle Scholar
  13. 13.
    Lukas, M.A. (1992): Methods for choosing the regularization parameter. In: R.S. Anderssen, A.K. Pani (eds.), Inverse problems in partial differential equations, Proceedings of the Centre for Mathematics and its Applications, Australian National UniversityGoogle Scholar
  14. 14.
    Nashed, M.Z., Wahba, G. (1974): Generalized inverses in reproducing kernel spaces: An approach to regularization of linear operator equations. SIAM J. Math. Anal.5, 974–987Google Scholar
  15. 15.
    Nychka, D.W., Cox, D.D. (1989): Convergence rates for regularized solutions of integral equations from discrete noisy data. Ann. Stat.17, 556–572Google Scholar
  16. 16.
    O'Sullivan, F. (1990): Convergence characteristics of methods of regularization estimators for non linear operator equations. SIAM J. Numer. Anal.27, 1635–1649Google Scholar
  17. 17.
    Ragozin, D. (1983): Error bounds for derivative estimates based on spline smoothing of exact or noisy data, J. Approx. Theory37, 335–355Google Scholar
  18. 18.
    Rice, J., Rosenblatt, M. (1981): Integrated mean squared error of a smoothing spline. J. Approx. Theory33, 353–369Google Scholar
  19. 19.
    Rice, J., Rosenblatt M. (1983): Smoothing splines: regression, derivatives and deconvolution. Ann. Stat.11, 141–156Google Scholar
  20. 20.
    Speckman, P. (1981): The asymptotic integrated mean square error for smoothing noisy data by splines. Unpublished manuscriptGoogle Scholar
  21. 21.
    Speckman, P. (1985): Spline smoothing and optimal rates of convergence in nonparametric regression. Ann. Stat.13, 970–983Google Scholar
  22. 22.
    Utreras, F. (1983): Natural spline functions, their associated eigenvalue problem. Numer. Math.42, 107–117Google Scholar
  23. 23.
    Utreras, F. (1987): On generalized cross-validation for multivariate smoothing spline functions. SIAM J. Sci. Stat. Comput.8, 630–643Google Scholar
  24. 24.
    Utreras, F. (1988): Convergence rates for multivariate smoothing spline functions. J. Approx. Theory52, 1–27Google Scholar
  25. 25.
    Wahba, G. (1977): Practical approximate solutions to linear operator equations when the data are noisy. SIAM J. Numer. Anal.14, 651–667Google Scholar
  26. 26.
    Wahba, G. (1979): Smoothing and ill-posed problems. In: Goldberg, M.A. (ed.) Solution methods for integral equations with applications, pp. 183–194. Plenum Press: New YorkGoogle Scholar
  27. 27.
    Wahba, G. (1985): A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem. Ann. Stat.13, 1378–1402Google Scholar
  28. 28.
    Wahba, G., Wang, Y. (1990): When is the optimal regularization parameter insensitive to the choice of the loss function? Commun. Stat. Theory Meth.19, 1685–1700Google Scholar
  29. 29.
    Wahba, G. (1990): Spline models for observational data, CBMS-NSF Regional Conference Series in Applied Mathematics 59, SIAM, PhiladelphiaGoogle Scholar

Copyright information

© Springer-Verlag 1993

Authors and Affiliations

  • Mark A. Lukas
    • 1
  1. 1.School of Mathematical and Physical SciencesMurdoch UniversityMurdochAustralia

Personalised recommendations