, Volume 22, Issue 2, pp 183–221 | Cite as

Model-free model-fitting and predictive distributions

Invited Paper


The problem of prediction is revisited with a view towards going beyond the typical nonparametric setting and reaching a fully model-free environment for predictive inference, i.e., point predictors and predictive intervals. A basic principle of model-free prediction is laid out based on the notion of transforming a given setup into one that is easier to work with, namely i.i.d. or Gaussian. As an application, the problem of nonparametric regression is addressed in detail; the model-free predictors are worked out, and shown to be applicable under minimal assumptions. Interestingly, model-free prediction in regression is a totally automatic technique that does not necessitate the search for an optimal data transformation before model fitting. The resulting model-free predictive distributions and intervals are compared to their corresponding model-based analogs, and the use of cross-validation is extensively discussed. As an aside, improved prediction intervals in linear regression are also obtained.


Bootstrap Cross-validation Frequentist prediction Heteroskedasticity Nonparametric estimation Prediction intervals Regression Smoothing Transformations 

Mathematics Subject Classification

62G99 62G08 62G09 


  1. Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185 MathSciNetGoogle Scholar
  2. Atkinson AC (1985) Plots, transformations and regression. Clarendon, Oxford MATHGoogle Scholar
  3. Beran R (1990) Calibrating prediction regions. J Am Stat Assoc 85:715–723 MathSciNetMATHCrossRefGoogle Scholar
  4. Bickel P, Li B (2006) Regularization in statistics. Test 15(2):271–344 MathSciNetMATHCrossRefGoogle Scholar
  5. Box GEP, Cox DR (1964) An analysis of transformations. J R Stat Soc, Ser B, Stat Methodol 26:211–252 MathSciNetMATHGoogle Scholar
  6. Breiman L, Friedman J (1985) Estimating optimal transformations for multiple regression and correlation. J Am Stat Assoc 80:580–597 MathSciNetMATHCrossRefGoogle Scholar
  7. Carmack PS, Schucany WR, Spence JS, Gunst RF, Lin Q, Haley RW (2009) Far casting cross-validation. J Comput Graph Stat 18(4):879–893 MathSciNetCrossRefGoogle Scholar
  8. Carroll RJ, Ruppert D (1988) Transformations and weighting in regression. Chapman & Hall, New York Google Scholar
  9. Carroll RJ, Ruppert D (1991) Prediction and tolerance intervals with transformation and/or weighting. Technometrics 33:197–210 MathSciNetCrossRefGoogle Scholar
  10. Cox DR (1975) Prediction intervals and empirical Bayes confidence intervals. In: Gani J (ed) Perspectives in probability and statistics. Academic Press, London, pp 47–55 Google Scholar
  11. Dai J, Sperlich S (2010) Simple and effective boundary correction for kernel densities and regression with an application to the world income and Engel curve estimation. Comput Stat Data Anal 54(11):2487–2497 MathSciNetCrossRefGoogle Scholar
  12. DasGupta A (2008) Asymptotic theory of statistics and probability. Springer, New York MATHGoogle Scholar
  13. Davison AC, Hinkley DV (1997) Bootstrap methods and their applications. Cambridge University Press, Cambridge Google Scholar
  14. Dawid AP (2004) Probability, causality, and the empirical world: a Bayes–de Finetti–Popper–Borel synthesis. Stat Sci 19(1):44–57 MathSciNetMATHCrossRefGoogle Scholar
  15. Draper NR, Smith H (1998) Applied regression analysis, 3rd edn. Wiley, New York MATHGoogle Scholar
  16. Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Stat 7:1–26 MathSciNetMATHCrossRefGoogle Scholar
  17. Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78:316–331 MathSciNetMATHCrossRefGoogle Scholar
  18. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York MATHGoogle Scholar
  19. Fan J, Gijbels I (1996) Local polynomial modelling and its applications. Chapman & Hall, London MATHGoogle Scholar
  20. Freedman DA (1981) Bootstrapping regression models. Ann Stat 9:1218–1228 MATHCrossRefGoogle Scholar
  21. Gangopadhyay AK, Sen PK (1990) Bootstrap confidence intervals for conditional quantile functions. Sankhya, Ser A 52(3):346–363 MathSciNetMATHGoogle Scholar
  22. Goldberger AS (1962) Best linear unbiased prediction in the generalized linear regression model. J Am Stat Assoc 57:369–375 MathSciNetMATHCrossRefGoogle Scholar
  23. Geisser S (1993) Predictive inference: an introduction. Chapman & Hall, New York MATHGoogle Scholar
  24. Hahn J (1995) Bootstrapping quantile regression estimators. Econom Theory 11(1):105–121 CrossRefGoogle Scholar
  25. Hall P (1992) The bootstrap and edgeworth expansion. Springer, New York Google Scholar
  26. Hall P (1993) On edgeworth expansion and bootstrap confidence bands in nonparametric curve estimation. J R Stat Soc, Ser B, Stat Methodol 55:291–304 MATHGoogle Scholar
  27. Hall P, Wehrly TE (1991) A geometrical method for removing edge effects from kernel type nonparametric regression estimators. J Am Stat Assoc 86:665–672 MathSciNetCrossRefGoogle Scholar
  28. Härdle W (1990) Applied nonparametric regression. Cambridge University Press, Cambridge MATHCrossRefGoogle Scholar
  29. Härdle W, Bowman AW (1988) Bootstrapping in nonparametric regression: local adaptive smoothing and confidence bands. J Am Stat Assoc 83:102–110 MATHGoogle Scholar
  30. Härdle W, Marron JS (1991) Bootstrap simultaneous error bars for nonparametric regression. Ann Stat 19:778–796 MATHCrossRefGoogle Scholar
  31. Hart JD (1997) Nonparametric smoothing and lack-of-fit tests. Springer, New York MATHCrossRefGoogle Scholar
  32. Hart JD, Yi S (1998) One-sided cross-validation. J Am Stat Assoc 93(442):620–631 MathSciNetMATHCrossRefGoogle Scholar
  33. Hong Y (1999) Hypothesis testing in time series via the empirical characteristic function: a generalized spectral density approach. J Am Stat Assoc 94:1201–1220 MATHCrossRefGoogle Scholar
  34. Hong Y, White H (2005) Asymptotic distribution theory for nonparametric entropy measures of serial dependence. Econometrica 73(3):837–901 MathSciNetMATHCrossRefGoogle Scholar
  35. Horowitz J (1998) Bootstrap methods for median regression models. Econometrica 66(6):1327–1351 MathSciNetMATHCrossRefGoogle Scholar
  36. Koenker R (2005) Quantile regression. Cambridge University Press, Cambridge MATHCrossRefGoogle Scholar
  37. Li Q, Racine JS (2007) Nonparametric econometrics. Princeton University Press, Princeton MATHGoogle Scholar
  38. Linton OB, Sperlich S, van Keilegom I (2008) Estimation of a semiparametric transformation model. Ann Stat 36(2):686–718 MATHCrossRefGoogle Scholar
  39. Loader C (1999) Local regression and likelihood. Springer, New York MATHGoogle Scholar
  40. McCullagh P, Nelder J (1983) Generalized linear models. Chapman & Hall, London MATHGoogle Scholar
  41. McMurry T, Politis DN (2008) Bootstrap confidence intervals in nonparametric regression with built-in bias correction. Stat Probab Lett 78:2463–2469 MathSciNetMATHCrossRefGoogle Scholar
  42. McMurry T, Politis DN (2010) Banded and tapered estimates of autocovariance matrices and the linear process bootstrap. J Time Ser Anal 31:471–482 MathSciNetMATHCrossRefGoogle Scholar
  43. Nadaraya EA (1964) On estimating regression. Theory Probab Appl 9:141–142 CrossRefGoogle Scholar
  44. Neumann M, Polzehl J (1998) Simultaneous bootstrap confidence bands in nonparametric regression. J Nonparametr Stat 9:307–333 MathSciNetMATHCrossRefGoogle Scholar
  45. Olive DJ (2007) Prediction intervals for regression models. Comput Stat Data Anal 51:3115–3122 MathSciNetMATHCrossRefGoogle Scholar
  46. Pagan A, Ullah A (1999) Nonparametric econometrics. Cambridge University Press, Cambridge Google Scholar
  47. Patel JK (1989) Prediction intervals: a review. Commun Stat, Theory Methods 18:2393–2465 MATHCrossRefGoogle Scholar
  48. Politis DN (2003) A normalizing and variance-stabilizing transformation for financial time series. In: Akritas MG, Politis DN (eds) Recent advances and trends in nonparametric statistics. Elsevier, Amsterdam, pp 335–347 CrossRefGoogle Scholar
  49. Politis DN (2007a) Model-free vs. model-based volatility prediction. J Financ Econom 5(3):358–389 MathSciNetGoogle Scholar
  50. Politis DN (2007b) Model-free prediction. In: Bulletin of the international statistical institute—volume LXII, Lisbon, 22–29 Aug 2007, pp 1391–1397 Google Scholar
  51. Politis DN (2010) Model-free model-fitting and predictive distributions. Discussion Paper, Department of Economics, Univ of California—San Diego. Retrieved from: http://escholarship.org/uc/item/67j6s174
  52. Politis DN, Romano JP, Wolf M (1999) Subsampling. Springer, New York MATHCrossRefGoogle Scholar
  53. Rosenblatt M (1952) Remarks on a multivariate transformation. Ann Math Stat 23:470–472 MathSciNetMATHCrossRefGoogle Scholar
  54. Ruppert D, Cline DH (1994) Bias reduction in kernel density estimation by smoothed empirical transformations. Ann Stat 22:185–210 MathSciNetMATHCrossRefGoogle Scholar
  55. Schmoyer RL (1992) Asymptotically valid prediction intervals for linear models. Technometrics 34:399–408 MathSciNetMATHCrossRefGoogle Scholar
  56. Schucany WR (2004) Kernel smoothers: an overview of curve estimators for the first graduate course in nonparametric statistics. Stat Sci 19:663–675 MathSciNetMATHCrossRefGoogle Scholar
  57. Seber GAF, Lee AJ (2003) Linear regression analysis. Wiley, New York MATHCrossRefGoogle Scholar
  58. Shao J, Tu D (1995) The jackknife and bootstrap. Springer, New York MATHCrossRefGoogle Scholar
  59. Shi SG (1991) Local bootstrap. Ann Inst Stat Math 43:667–676 MATHCrossRefGoogle Scholar
  60. Stine RA (1985) Bootstrap prediction intervals for regression. J Am Stat Assoc 80:1026–1031 MathSciNetMATHCrossRefGoogle Scholar
  61. Tibshirani R (1988) Estimating transformations for regression via additivity and variance stabilization. J Am Stat Assoc 83:394–405 MathSciNetCrossRefGoogle Scholar
  62. Tibshirani R (1996) Regression shrinkage and selection via the Lasso. J R Stat Soc, Ser B, Stat Methodol 58(1):267–288 MathSciNetMATHGoogle Scholar
  63. Wang L, Brown LD, Cai TT, Levine M (2008) Effect of mean on variance function estimation in nonparametric regression. Ann Stat 36:646–664 MathSciNetMATHCrossRefGoogle Scholar
  64. Watson GS (1964) Smooth regression analysis. Sankhya, Ser A 26:359–372 MathSciNetMATHGoogle Scholar
  65. Wolfowitz J (1957) The minimum distance method. Ann Math Stat 28:75–88 MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Sociedad de Estadística e Investigación Operativa 2013

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of California—San DiegoLa JollaUSA

Personalised recommendations