# Asymptotic optimality of generalized cross-validation for choosing the regularization parameter

- 230 Downloads
- 31 Citations

## Summary

Let*f*_{ n }*λ* be the regularized solution of a general, linear operator equation,*Kf*_{ 0 }*=g*, from discrete, noisy data*y*_{ i }=*g*(*x*) +*ɛ*_{ i },*i*=1,...,*n*, where*ɛ*_{ i } are uncorrelated random errors. We consider the prominent method of generalized cross-validation (GCV) for choosing the crucial regularization parameter λ. The practical GCV estimate\(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\lambda } _V \) and its “expected” counterpart λ_{ V } are defined as the minimizers of the GCV function*V*(λ) and*EV*(λ), respectively, where*E* denotes expectation. We investigate the asymptotic performance of λ_{ V } with respect to each of the following loss functions: the risk, an*L*^{2} norm on the output error*Kf*_{ n }λ−*g*, and a whole class of stronger norms on the input error*f*_{ n }λ−*f*_{0}. In the special cases of data smoothing and Fourier differentiation, it is known that as*n*→∞, λ_{ V } is asymptotically optimal (ao) with respect to the risk criterion. We show this to be true in general, and also extend it to the*L*^{2} norm criterion. The asymptotic optimality is independent of the error variance, the ill-posedness of the problem and the smoothness index of the solution*f*_{0}. For the input error criterion, it is shown that λ_{ V } is weakly ao for a certain class of*f*_{0} if the smoothness of*f*_{0} relative to the regularization space is not too high, but otherwise λ_{ V } is sub-optimal. This result is illustrated in the case of numerical differentiation.

## Mathematics Subject Classifications (1991)

65J10 62G05 47A50## Preview

Unable to display preview. Download preview PDF.

## References

- 1.Cox, D.D. (1983): Asymptotics for M-type smoothing splines. Ann. Stat.
**11**, 530–551Google Scholar - 2.Cox, D.D. (1984): Multivariate smoothing spline functions. SIAM J. Numer. Anal.
**21**, 789–813Google Scholar - 3.Cox, D.D. (1988): Approximation of method of regularization estimators. Ann. Stat.
**16**, 694–712Google Scholar - 4.Craven, P., Wahba, G. (1979): Smoothing noisy data with spline functions. Numer. Math.
**31**, 377–403Google Scholar - 5.Davies, A.R., Anderssen, R.S. (1986): Improved estimates of statistical regularization parameters in Fourier differentiation and smoothing. Numer. Math.
**48**, 671–697Google Scholar - 6.Davies, A.R., Anderssen, R.S. (1986): Optimization in the regularization of ill-posed problems. J. Austral. Math. Soc. Ser. B
**28**, 114–133Google Scholar - 7.Golub, G.H., Heath, M., Wahba, G. (1979): Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics
**21**, 215–223Google Scholar - 8.Kimeldorf, G.S., Wahba, G. (1971): Some results on Tchebycheffian spline functions. J. Math. Anal. Appl.
**33**, 82–95Google Scholar - 9.Li, K.C. (1986): Asymptotic optimality of
*C*_{L}and generalized cross-validation in ridge regression with application to spline smoothing. Ann. Stat.**14**, 1101–1112Google Scholar - 10.Lukas, M.A. (1988): Convergence rates for regularized solutions. Math. Comp.
**51**, 107–131Google Scholar - 11.Lukas, M.A. (1988): Assessing regularized solutions. J. Austral. Math. Soc. Ser. B
**30**, 24–42Google Scholar - 12.Lukas, M.A. (1991): Asymptotic behaviour of the discrepancy principle and generalized maximum likelihood for choosing the regularization parameter. Report No. 4, Centre for Mathematics and its Applications, Australian National UniversityGoogle Scholar
- 13.Lukas, M.A. (1992): Methods for choosing the regularization parameter. In: R.S. Anderssen, A.K. Pani (eds.), Inverse problems in partial differential equations, Proceedings of the Centre for Mathematics and its Applications, Australian National UniversityGoogle Scholar
- 14.Nashed, M.Z., Wahba, G. (1974): Generalized inverses in reproducing kernel spaces: An approach to regularization of linear operator equations. SIAM J. Math. Anal.
**5**, 974–987Google Scholar - 15.Nychka, D.W., Cox, D.D. (1989): Convergence rates for regularized solutions of integral equations from discrete noisy data. Ann. Stat.
**17**, 556–572Google Scholar - 16.O'Sullivan, F. (1990): Convergence characteristics of methods of regularization estimators for non linear operator equations. SIAM J. Numer. Anal.
**27**, 1635–1649Google Scholar - 17.Ragozin, D. (1983): Error bounds for derivative estimates based on spline smoothing of exact or noisy data, J. Approx. Theory
**37**, 335–355Google Scholar - 18.Rice, J., Rosenblatt, M. (1981): Integrated mean squared error of a smoothing spline. J. Approx. Theory
**33**, 353–369Google Scholar - 19.Rice, J., Rosenblatt M. (1983): Smoothing splines: regression, derivatives and deconvolution. Ann. Stat.
**11**, 141–156Google Scholar - 20.Speckman, P. (1981): The asymptotic integrated mean square error for smoothing noisy data by splines. Unpublished manuscriptGoogle Scholar
- 21.Speckman, P. (1985): Spline smoothing and optimal rates of convergence in nonparametric regression. Ann. Stat.
**13**, 970–983Google Scholar - 22.Utreras, F. (1983): Natural spline functions, their associated eigenvalue problem. Numer. Math.
**42**, 107–117Google Scholar - 23.Utreras, F. (1987): On generalized cross-validation for multivariate smoothing spline functions. SIAM J. Sci. Stat. Comput.
**8**, 630–643Google Scholar - 24.Utreras, F. (1988): Convergence rates for multivariate smoothing spline functions. J. Approx. Theory
**52**, 1–27Google Scholar - 25.Wahba, G. (1977): Practical approximate solutions to linear operator equations when the data are noisy. SIAM J. Numer. Anal.
**14**, 651–667Google Scholar - 26.Wahba, G. (1979): Smoothing and ill-posed problems. In: Goldberg, M.A. (ed.) Solution methods for integral equations with applications, pp. 183–194. Plenum Press: New YorkGoogle Scholar
- 27.Wahba, G. (1985): A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem. Ann. Stat.
**13**, 1378–1402Google Scholar - 28.Wahba, G., Wang, Y. (1990): When is the optimal regularization parameter insensitive to the choice of the loss function? Commun. Stat. Theory Meth.
**19**, 1685–1700Google Scholar - 29.Wahba, G. (1990): Spline models for observational data, CBMS-NSF Regional Conference Series in Applied Mathematics 59, SIAM, PhiladelphiaGoogle Scholar