Sankhya A

, Volume 75, Issue 1, pp 74–95 | Cite as

Mixing partially linear regression models



In semiparametric modeling, often the main interest is on the parametric part. When a model selection step is performed to choose important variables for the parametric part, the selection uncertainty may or may not be negligible. If it is high, not only the identified model is not trustworthy but also the resulting estimator may have unnecessarily high variability. We propose model selection diagnostic measures both for variable selection and for regression estimation, which provide crucial information on reliability of the selected model and choice between model selection and model averaging for regression estimation. We study a model mixing method for estimating the linear function in a partially linear regression model and derive oracle inequalities that show the mixed estimators perform nearly as well as the best estimator from the candidate models. Simulation and data examples are encouraging.

Keywords and phrases

Adaptive regression by mixing Model selection Model selection diagnostics Partially linear model Semi-parametric modeling 

AMS (2000) subject classification

Primary 62G08 62G10 Secondary 62G20 


  1. barron, a.r. (1987). Are Bayes Rules Consistent in Information? In Open Problems in Communication and Computation, (T.M. Cover and B. Gopinath, eds.). Springer, Berlin, pp 85–91.CrossRefGoogle Scholar
  2. bickel, p.j., klaassen, c.a.j., ritov, y. and wellner, j.a. (1993). Efficient and adaptive estimation for semiparametric models. John Hopkins Univ. Press, Baltimore.MATHGoogle Scholar
  3. breiman, l. (1996). Bagging predictors. Mach. Learn., 24, 123–140.MathSciNetMATHGoogle Scholar
  4. breiman, l. (2001). Statistical modeling: the two cultures. Statist. Sci., 16, 199–231.MathSciNetMATHCrossRefGoogle Scholar
  5. buckland, s.t., burnham, k.p. and augustin, n.h. (1997). Model selection: an integral part of inference. Biometrics, 53, 603–618.MATHCrossRefGoogle Scholar
  6. bunea, f. (2004). Consistent covariate selection and post model selection inference in semiparametric regression. Ann. Statist., 32, 898–927.MathSciNetMATHCrossRefGoogle Scholar
  7. bunea, f. and nobel, a. (2008). Sequential procedures for aggregating arbitrary estimators of a conditional mean. IEEE Trans. Inform. Theory, 54, 1725–1735.MathSciNetCrossRefGoogle Scholar
  8. bunea, f. and wegkamp, m.h. (2004). Two-stage model selection procedures in partially linear regression. Canad. J. Statist., 32, 1–14.MathSciNetCrossRefGoogle Scholar
  9. burnham, k.p. and anderson, d.r. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociol. Methods Res., 33, 261–304.MathSciNetCrossRefGoogle Scholar
  10. catoni, o. (2004). Statistical learning theory and stochastic optimization. Springer, New York.MATHCrossRefGoogle Scholar
  11. chen, h. and chen, k.-w. (1991). Selection of the splined variables and convergence rates in a partial spline model. Canad. J. Statist., 19, 323–339.MathSciNetMATHCrossRefGoogle Scholar
  12. chen, h. and shiau, j.h. (1991). A two-stage spline smoothing method for partially linear models. J. Statist. Plann. Inference, 27, 187–201.MathSciNetMATHCrossRefGoogle Scholar
  13. chen, h. and shiau, j.h. (1994). Data-driven efficient estimators for a partially linear model. Ann. Statist., 22, 211–237.MathSciNetMATHCrossRefGoogle Scholar
  14. chen, l., giannakouros, p. and yang, y. (2007). Model combining in factorial data analysis. J. Statist. Plann. Inference, 137, 2920–2934.MathSciNetMATHCrossRefGoogle Scholar
  15. dalalyan, a.s. and tsybakov, a.b. (2007). Aggregation by Exponential Weighting and Sharp Oracle Inequalities. In Lecture Notes in Computer Science 4539 LNAI, pp 97–111.Google Scholar
  16. hjort, n.l. and claeskens, g. (2003). Frequentist model average estimators. J. Amer. Statist. Assoc., 98, 879–899.MathSciNetMATHCrossRefGoogle Scholar
  17. engle, r.f., granger, c.w., rice, j. and weiss, a. (1986). Semiparametric estimates of the relation between weather and electricity sales. J. Amer. Statist. Assoc., 81, 310–320.CrossRefGoogle Scholar
  18. goldenshluger, a. (2009). A universal procedure for aggregating estimators. Ann. Statist., 37, 542–568.MathSciNetMATHCrossRefGoogle Scholar
  19. hastie, t. and tibshirani, r. (1990). Generalized additive models. Chapman and Hall, London.MATHGoogle Scholar
  20. härdle, w., liang, h. and gao, j. (2000). Partially linear models. Springer, Berlin.MATHCrossRefGoogle Scholar
  21. härdle, w., müller, m., sperlich, s. and werwarts, a. (2004). Nonparametric and semiparametric models. Springer, Berlin.MATHCrossRefGoogle Scholar
  22. hoeting, j., madigan, d., raftery, a. and volinsky, c. (1999). Bayesian model averaging: a tutorial (with discussion). Statist. Sci., 14, 382–417.MathSciNetMATHCrossRefGoogle Scholar
  23. leung, g. and barron, a.r. (2006). Information theory and mixing least-squares regressions. IEEE Trans. Inform. Theory, 52, 3396–3410.MathSciNetCrossRefGoogle Scholar
  24. liang, h. and li, r. (2009). Variable selection for partially linear models with measurement errors. J. Amer. Statist. Assoc., 104, 234–248.MathSciNetCrossRefGoogle Scholar
  25. liu, s. and yang, y. (2012). Combining models in longitudinal data analysis. Ann. Inst. Statist. Math., 64, 233–254.MathSciNetMATHCrossRefGoogle Scholar
  26. ni, x., zhang, h.h. and zhang d. (2009). Automatic model selection for partially linear models. J. Multivariate Anal., 100, 2100–2111.MathSciNetMATHCrossRefGoogle Scholar
  27. ruppert, d., wand, m.p. and carroll, r.j. (2003). Semiparametric regression. Cambridge University Press, Cambridge.MATHCrossRefGoogle Scholar
  28. shen, x. and huang, h.-c. (2006). Optimal model assessment, selection, and combination. J. Amer. Statist. Assoc., 101, 554–568.MathSciNetMATHCrossRefGoogle Scholar
  29. shiau, j., wahba, g. and johnson, d.r. (1986). Partial spline models for the inclusion of tropopause and frontal boundary information in otherwise smooth two and three dimensional objective analysis. J. Atmos. Oceanic Technol., 3, 713–725.CrossRefGoogle Scholar
  30. simonoff, j.s. and tsai, c. (1999). Semiparametric and Additive Model Selection Using an Improved Akaike Information Criterion, J. Comput. Graph. Statist., 8, 22–40.MathSciNetGoogle Scholar
  31. speckman, p. (1988). Kernel smoothing in partial linear models. J. R. Stat. Soc. Ser. B, 50, 413–436.MathSciNetMATHGoogle Scholar
  32. wahba, g. (1984). Partial Spline Models for the Semiparametric Estimation of Functions of Several Variables. In Statistical Analysis of Time Series. Institute of Statistical Mathematics, Tokyo, pp 312–329.Google Scholar
  33. wang, h., li, r. and tsai, c.l. (2007). Tuning parameter selector for SCAD. Biometrika, 94, 553–568.MathSciNetMATHCrossRefGoogle Scholar
  34. yang, y. (2001). Adaptive regression by mixing. J. Amer. Statist. Assoc., 96, 574–588.MathSciNetMATHCrossRefGoogle Scholar
  35. yang, y. (2007). Consistency of cross validation for comparing regression procedures. Ann. Statist., 35, 2450–2473.MathSciNetMATHCrossRefGoogle Scholar
  36. yuan, z. and yang, y. (2005). Combining linear regression models: when and how? J. Amer. Statist. Assoc., 100, 1202–1214.MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Indian Statistical Institute 2013

Authors and Affiliations

  1. 1.Consumer Banking JPMorgan ChaseColumbusUSA
  2. 2.School of StatisticsThe University of MinnesotaMinneapolisUSA

Personalised recommendations