Abstract
We consider model selection and estimation for partial spline models and propose a new regularization method in the context of smoothing splines. The regularization method has a simple yet elegant form, consisting of roughness penalty on the nonparametric component and shrinkage penalty on the parametric components, which can achieve function smoothing and sparse estimation simultaneously. We establish the convergence rate and oracle properties of the estimator under weak regularity conditions. Remarkably, the estimated parametric components are sparse and efficient, and the nonparametric component can be estimated with the optimal rate. The procedure also has attractive computational properties. Using the representer theory of smoothing splines, we reformulate the objective function as a LASSO-type problem, enabling us to use the LARS algorithm to compute the solution path. We then extend the procedure to situations when the number of predictors increases with the sample size and investigate its asymptotic properties in that context. Finite-sample performance is illustrated by simulations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abramowitz, M., Stegun, I. (1964). Handbook of mathematical functions with formulas, graphs, and mathematical tables. New York: Dover.
Breiman, L. (1995). Better subset selection using the nonnegative garrote. Technometrics, 37, 373–384.
Bickel, P.J., Ritov, Y., Tsybakov, A.B. (2009). Simultaneous analysis of Lasso and Dantzig selector. Annals of Statistics, 37, 1705–1732.
Craven, P., Wahba, G. (1979). Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation. Numerishe Mathematik, 31, 377–403.
Denby, L. (1984). Smooth regression functions. Ph.D. Thesis. Department of Statistics. University of Michigan.
Efron, B., Hastie, T., Johnstone, I., Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32, 407–451.
Fan, J., Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of American Statistical Association, 96, 1348–1360.
Fan, J., Li, R. (2004). New estimation and model selection procedures for semiparametric modeling in longitudinal data analysis. Journal of American Statistical Association, 99, 710–723.
Fan, J., Lv, J. (2008). Sure independence screening for ultra-high dimensional feature space. Journal of Royal Statistical Society B (with discussion) , 70, 849–911.
Fan, J., Peng, H. (2004). Nonconcave penalized likelihood with a diverging number of parameters. Annals of Statistics, 32, 928–961.
Green, P.J., Silverman, B.W. (1994). Nonparametric regression and generalized linear models. London: Chapman and Hall.
Gu, C. (2002). Smoothing spline ANOVA models. New York: Springer.
Heckman, N. (1986). Spline smoothing in a partly linear models. Journal of Royal Statistical Society Series B, 48, 244–248.
Huang, J., Horowitz, J., Ma, S. (2008a). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Annals of Statistics, 36, 587–613.
Huang, J., Ma, S., Zhang, C.H. (2008b). Adaptive LASSO for sparse high dimensional regression. Statistica Sinica, 18, 1603–1618.
Kimeldorf, G., Wahba, G. (1971). Some results on Tchebycheffian spline functions. Journal of Mathematical Analysis and Applications, 33, 82–95.
Mammen, E., van de Geer, S. (1997). Penalized quasi-likelihood estimation in partially linear models. Annals of Statistics, 25, 1014–1035.
Ni, X., Zhang, H.H., Zhang, D. (2009). Automatic model selection for partially linear models. Journal of Multivariate Analysis, 100, 2100–2111.
Portnoy, S. (1984). Asymptotic behavior of M-estimator of \(p\) regression parameters when \(p^2/n\) is large. I. Consistency. Annals of Statistics, 12, 1298–1309.
Rice, J. (1986). Convergence rates for partially spline model. Statistics and Probability Letters, 4, 203–208.
Ruppert, D., Wand, M.P., Carroll, R.J. (2003). Semiparametric regression. Cambridge: Cambridge University Press.
Shang, Z., Cheng, G. (2013). Local and global asymptotic inference in smoothing spline models. Annals of Statistics, (to appear).
Shao, J. (2003). Mathematical statistics (2nd ed.). New York: Springer.
Shiau, J., Wahba, G. (1988). Rates of convergence for some estimates of a semi-parametric model. Communications in Statistics Simulation and Computation, 17, 111–113.
Speckman, P. (1988). Kernel smoothing in partially linear models. Journal of Royal Statistical Society-B, 50, 413–436.
Stamey, T., Kabalin, J., McNeal, J., Johnstone, I., Freida, F., Redwine, E., et al. (1989). Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate II radical prostatectomy treated patients. Journal of Urology, 16, 1076–1083.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B, 58, 267–288.
van der Vaart, A. W., Wellner, J.A. (1996). Weak convergence and empirical processes: with applications to statistics. New York: Springer.
Wahba, G. (1984) Partial spline models for the semiparametric estimation functions of several variables. In H.A. David, H. T. David (Eds.), Statistics: An appraisal, proceedings of the 50th anniversary conference. Ames: Iowa State University Press.
Wahba, G. (1990), Spline models for observational data. SIAM. CBMS-NSF, Regional Conference Series in Applied Mathematics, Vol. 59. Philadelphia.
Wang, H., Li, R., Tsai, C.L. (2007a). Tuning parameter selectors for the smoothly clipped absolute deviation method. Biometrika, 94, 553–568.
Wang, H., Li, G., Jiang, G. (2007b). Robust regression shrinkage and consistent variable selection via the LAD-LASSO. Journal of Business & Economics Statistics, 20, 347–355.
Wang, H., Li, B., Leng, C. (2009). Shrinkage tuning parameter selection with a diverging number of parameters. Journal of Royal Statistical Society, Series B, 71, 671–683.
Yatchew, A. (1997). An elementary estimator of the partial linear model. Economics Letters, 57, 135–143.
Zhang, H.H., Lu, W. (2007). Adaptive-LASSO for Cox’s proportional hazards model. Biometrika, 94, 691–703.
Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of American Statistical Association, 101, 1418–1429.
Zou, H., Zhang, H.H. (2009). On the adaptive elastic-net with a diverging number of parameters. Annals of Statistics, 37, 1733–1751.
Author information
Authors and Affiliations
Corresponding author
Additional information
G. Cheng: Supported by NSF Grant DMS-0906497 and CAREER Award DMS-1151692. H. H. Zhang: Supported by NSF grants DMS-0645293, DMS-1347844, NIH grants P01 CA142538 and R01 CA085848.
About this article
Cite this article
Cheng, G., Zhang, H.H. & Shang, Z. Sparse and efficient estimation for partial spline models with increasing dimension. Ann Inst Stat Math 67, 93–127 (2015). https://doi.org/10.1007/s10463-013-0440-y
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10463-013-0440-y