Pointwise convergence in probability of general smoothing splines

  • Matthew ThorpeEmail author
  • Adam M. Johansen


Establishing the convergence of splines can be cast as a variational problem which is amenable to a \(\Gamma \)-convergence approach. We consider the case in which the regularization coefficient scales with the number of observations, n, as \(\lambda _n=n^{-p}\). Using standard theorems from the \(\Gamma \)-convergence literature, we prove that the general spline model is consistent in that estimators converge in a sense slightly weaker than weak convergence in probability for \(p\le \frac{1}{2}\). Without further assumptions, we show this rate is sharp. This differs from rates for strong convergence using Hilbert scales where one can often choose \(p>\frac{1}{2}\).


Variational methods \(\Gamma \)-convergence Pointwise convergence General spline model Nonparametric smoothing 



This work was carried out whilst MT was part of MASDOC at the University of Warwick and supported by an EPSRC Industrial CASE Award Ph.D. Studentship with Selex ES Ltd.


  1. Aerts, M., Claeskens, G., Wand, M. P. (2002). Some theory for penalized spline generalized additive models. Journal of Statistical Planning and Inference, 103(1–2), 455–470.Google Scholar
  2. Agapiou, S., Larsson, S., Stuart, A. M. (2013). Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems. Stochastic Processes and their Applications, 123(10), 3828–3860.Google Scholar
  3. Bissantz, N., Hohage, T., Munk, A. (2004). Consistency and rates of convergence of nonlinear Tikhonov regularization with random noise. Inverse Problems, 20(6), 1773–1789.Google Scholar
  4. Bissantz, N., Hohage, T., Munk, A., Ruymgaart, F. (2007). Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM Journal on Numerical Analysis, 45(6), 2610–2636.Google Scholar
  5. Bogachev, V. I. (1998). Gaussian measures. Providence: The American Mathematical Society.CrossRefzbMATHGoogle Scholar
  6. Braides, A. (2002). \(\Gamma \) -convergence for beginners. Oxford: Oxford University Press.Google Scholar
  7. Brown, L. D., Low, M. G. (1996). Asymptotic equivalence of nonparametric regression and white noise. The Annals of Statistics, 24(6), 2384–2398.Google Scholar
  8. Carroll, R. J., Van Rooij, A. C. M., Ruymgaart, F. H. (1991). Theoretical aspects of ill-posed problems in statistics. Acta Applicandae Mathematica, 24(2), 113–140.Google Scholar
  9. Claeskens, G., Krivobokova, T., Opsomer, J. D. (2009). Asymptotic properties of penalized spline estimators. Biometrika, 96(3), 529–544.Google Scholar
  10. Cox, D. D. (1983). Asymptotics for \(M\)-type smoothing splines. The Annals of Statistics, 11(2), 530–551.MathSciNetCrossRefzbMATHGoogle Scholar
  11. Cox, D. D. (1988). Approximation of method of regularization estimators. The Annals of Statistics, 16(2), 694–712.MathSciNetCrossRefzbMATHGoogle Scholar
  12. Craven, P., Wahba, G. (1979). Smoothing noisy data with spline functions. Numerische Mathematik, 31(4), 377–403.Google Scholar
  13. Dudley, R. M. (2002). Real analysis and probability. Cambridge: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  14. Eilers, P. H. C., Marx, B. D. (1996). Flexible smoothing with B-splines and penalties. Statistical Science, 11(2), 89–121.Google Scholar
  15. Feinberg, E. A., Kasyanov, P. O., Zadoianchuk, N. V. (2014). Fatou’s lemma for weakly converging probabilities. Theory of Probability & Its Applications, 58(4), 683–689.Google Scholar
  16. Goldenshluger, A., Pereverzev, S. V. (2000). Adaptive estimation of linear functionals in Hilbert scales from indirect white noise observations. Probability Theory and Related Fields, 118(2), 169–186.Google Scholar
  17. Hall, P., Opsomer, J. D. (2005). Theory for penalised spline regression. Biometrika, 92(1), 105–118.Google Scholar
  18. Hastie, T. J., Tibshirani, R. J. (1990). Generalized additive models. Boca Raton: Chapman and Hall.Google Scholar
  19. Hurvich, C. M., Simonoff, J. S., Tsai, C.-L. (1998). Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(2), 271–293.Google Scholar
  20. Kauermann, G., Krivobokova, T., Fahrmeir, L. (2009). Some asymptotic results on generalized penalized spline smoothing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71(2), 487–503.Google Scholar
  21. Kimeldorf, G. S., Wahba, G. (1970). A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. The Annals of Mathematical Statistics, 41(2), 495–502.Google Scholar
  22. Kimeldorf, G. S., Wahba, G. (1971). Some results on Tchebycheffian spline functions. Journal of Mathematical Analysis and Applications, 33(1), 82–95.Google Scholar
  23. Kou, S. C., Efron, B. (2002). Smoothers and the \(C_p\), generalized maximum likelihood, and extended exponential criteria: A geometric approach. Journal of the American Statistical Association, 97(459), 766–782.Google Scholar
  24. Lai, M.-J., Wang, L. (2013). Bivariate penalized splines for regression. Statistica Sinica, 23, 1399–1417.Google Scholar
  25. Li, K.-C. (1987). Asymptotic optimality for \(C_p\), \(C_L\), cross-validation and generalized cross-validation: Discrete index set. The Annals of Statistics, 15(3), 958–975.MathSciNetCrossRefzbMATHGoogle Scholar
  26. Li, Y., Ruppert, D. (2008). On the asymptotics of penalized splines. Biometrika, 95(2), 415–436.Google Scholar
  27. Lukas, M. A. (2006). Robust generalized cross-validation for choosing the regularization parameter. Inverse Problems, 22(5), 1883–1902.MathSciNetCrossRefzbMATHGoogle Scholar
  28. Mair, B. A., Ruymgaart, F. H. (1996). Statistical inverse estimation in Hilbert scales. SIAM Journal on Applied Mathematics, 56(5), 1424–1444.Google Scholar
  29. Mallows, C. L. (1973). Some comments on \(C_P\). Technometrics, 15(4), 661–675.zbMATHGoogle Scholar
  30. Nychka, D. W., Cox, D. D. (1989). Convergence rates for regularized solutions of integral equations from discrete noisy data. The Annals of Statistics, 17(2), 556–572.Google Scholar
  31. Ragozin, D. L. (1983). Error bounds for derivative estimates based on spline smoothing of exact or noisy data. Journal of Approximation Theory, 37(4), 335–355.MathSciNetCrossRefzbMATHGoogle Scholar
  32. Sakamoto, Y., Ishiguro, M., Kitagawa, G. (1986). Akaike information criterion statistics. Tokyo: KTK Scientific Publishers (KTK).Google Scholar
  33. Shen, J., Wang, X. (2011). Estimation of monotone functions via P-splines: A constrained dynamical optimization approach. SIAM Journal on Control and Optimization, 49(2), 646–671.Google Scholar
  34. Silverman, B. W. (1984). Spline smoothing: The equivalent variable kernel method. The Annals of Statistics, 12(3), 898–916.MathSciNetCrossRefzbMATHGoogle Scholar
  35. Speckman, P. L. (1985). Spline smoothing and optimal rates of convergence in nonparametric regression models. The Annals of Statistics, 13(3), 970–983.MathSciNetCrossRefzbMATHGoogle Scholar
  36. Speckman, P. L., Sun, D. (2001). Asymptotic properties of smoothing parameter selection in spline smoothing, Technical report, Department of Statistics, University of Missouri.
  37. Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. The Annals of Statistics, 10(4), 1040–1053.MathSciNetCrossRefzbMATHGoogle Scholar
  38. Thorpe, M., Johansen, A. M. (2016). Convergence and rates for fixed-interval multiple-track smoothing using \(k\)-means type optimization. Electronic Journal of Statistics, 10(2), 3693–3722.Google Scholar
  39. Thorpe, M., Theil, F., Johansen, A. M., Cade, N. (2015). Convergence of the \(k\)-means minimization problem using \(\Gamma \)-convergence. SIAM Journal on Applied Mathematics, 75(6), 2444–2474.Google Scholar
  40. Utreras, F. I. (1981). Optimal smoothing of noisy data using spline functions. SIAM Journal on Scientific and Statistical Computing, 2(3), 349–362.MathSciNetCrossRefzbMATHGoogle Scholar
  41. Utreras, F. I. (1983). Natural spline functions, their associated eigenvalue problem. Numerische Mathematik, 42(1), 107–117.MathSciNetCrossRefzbMATHGoogle Scholar
  42. Utreras, F. I. (1985). Smoothing noisy data under monotonicity constraints existence, characterization and convergence rates. Numerische Mathematik, 47(4), 611–625.MathSciNetCrossRefzbMATHGoogle Scholar
  43. Wahba, G. (1985). A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem. The Annals of Statistics, 13(4), 1378–1402.MathSciNetCrossRefzbMATHGoogle Scholar
  44. Wahba, G. (1990). Spline models for observational data. Philadelphia: Society for Industrial and Applied Mathematics (SIAM).CrossRefzbMATHGoogle Scholar
  45. Wahba, G., Wold, S. (1975). A completely automatic French curve: Fitting spline functions by cross-validation. Communications in Statistics, 4(1), 1–17.Google Scholar
  46. Wand, M. P. (1999). On the optimal amount of smoothing in penalised spline regression. Biometrika, 86(4), 936–940.MathSciNetCrossRefzbMATHGoogle Scholar
  47. Wang, X., Shen, J., Ruppert, D. (2011). On the asymptotics of penalized spline smoothing. Electronic Journal of Statistics, 5, 1–17.Google Scholar
  48. Xiao, L., Li, Y., Apanasovich, T. V., Ruppert, D. (2012). Local asymptotics of P-splines. arXiv:1201.0708
  49. Xiao, L., Li, Y., Ruppert, D. (2013). Fast bivariate P-splines: The sandwich smoother. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 75(3), 577–599.Google Scholar
  50. Yoshida, T., Naito, K. (2012). Asymptotics for penalized additive \(B\)-spline regression. Journal of the Japan Statistical Society, 42(1), 81–107.Google Scholar
  51. Yoshida, T., Naito, K. (2014). Asymptotics for penalised splines in generalised additive models. Journal of Nonparametric Statistics, 26(2), 269–289.Google Scholar

Copyright information

© The Institute of Statistical Mathematics, Tokyo 2017

Authors and Affiliations

  1. 1.Department of MathematicsCarnegie Mellon UniversityPittsburghUSA
  2. 2.Department of StatisticsUniversity of WarwickCoventryUK

Personalised recommendations