Nonparametric Estimation of Irregular Functions with Independent or Autocorrelated Errors

  • Michael Smith
  • Robert Kohn
Part of the Lecture Notes in Statistics book series (LNS, volume 133)


This paper shows how to estimate non-smooth univariate functions that are either discontinuous or have discontinuities in the first or higher order derivatives. The estimation is carried out by modelling the function as a linear combination of terms from one of a variety of nonorthonormal bases, such that each basis has elements that are themselves non-smooth. The advantage of such an approach over wavelet based alternatives is that the sample size need not be a power of two, nor do the observations of the independent variable have to be equally spaced. To estimate the nonparametric regression we use a hierarchical Bayesian framework, with the function estimate calculated using its posterior mean. To efficiently capture any irregularities in the functions we use bases with approximately as many elements as there are observations. The computation is made tractable by using a ‘focused’ Markov chain Monte Carlo sampling scheme. Both the hierarchical Bayesian model and the focused sampling scheme are extended to the case where the errors in a regression are potentially autocorrelated. The efficiency of the resulting estimators for both the independent error case and the autocorrelated case is demonstrated using simulation experiments based on several functions commonly examined in the literature.


Markov Chain Monte Carlo Gibbs Sampler Nonparametric Estimation Nonparametric Regression High Order Derivative 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Barnett, G., Kohn, R. and Sheather, S. (1996). Bayesian estimation of an autoregressive model using Markov Chain Monte Carlo. Journal of Econometrics.,74 237–254.MathSciNetMATHCrossRefGoogle Scholar
  2. Box, G. and Tiao, G. (1973). Bayesian Inference in Statistical Analysis. New York: John Wiley and Sons.MATHGoogle Scholar
  3. Clyde, M., Desimone, H. and Parmigiani, G. (1996). Prediction Via Orthogonalized Model Mixing. Journal of the American Statistical Association, 91 1197–1208MATHCrossRefGoogle Scholar
  4. Denison, D. Mallick, B. and Smith, A. (1996). Automatic Bayesian curve fitting. preprintGoogle Scholar
  5. Donoho, D. and Johnstone, I. (1994). Ideal Spatial Adaptation by Wavelet Shrinkage. Biometrika, 81 425–455.MathSciNetMATHCrossRefGoogle Scholar
  6. Donoho, D. and Johnstone, I. (1995). Adapting to Unknown Smoothness via Wavelet Shrinkage’. Journal of the Royal Statistical Society, Ser. B, 57 301–370.MathSciNetMATHGoogle Scholar
  7. Friedman, J. (1991). Multivariate adaptive regression splines. The Annals of Statistics (with discussion), 19 1–141MATHCrossRefGoogle Scholar
  8. Friedman, J. and Silverman, B. (1989). Flexible parsimonious smoothing and additive modelling. Technometrics, 31 3–39.MathSciNetMATHCrossRefGoogle Scholar
  9. George, E., and McCulloch, R. (1993). Variable selection via Gibbs sampling. Journal of the American Statistical Association, 88 881–889CrossRefGoogle Scholar
  10. Green, P. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82 711–732MathSciNetMATHCrossRefGoogle Scholar
  11. Hall, P. and Patil, P. (1996). On the choice of the smoothing parameter, threshold and truncation in nonparametric regression by nonlinear wavelet methods. Journal of the Royal Statistical Society, Series B, 58 361–377MathSciNetMATHGoogle Scholar
  12. Hall, P. and Nason, G. (1997). On choosing a non-integer resolution level when using wavelet methods. preprintGoogle Scholar
  13. Luo, Z. and Wahba, G., (1997). Hybrid Adaptive Splines. Journal of the American Statistical Association, 92 107–116.MathSciNetMATHCrossRefGoogle Scholar
  14. Mitchell, T., and Beauchamp J. (1988). Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83 1023–1036.MathSciNetMATHCrossRefGoogle Scholar
  15. Monahan, J. (1984). Full Bayesian analysis of ARMA time series models. Journal of Econometrics, 21 307–31.CrossRefGoogle Scholar
  16. O’Hagan, A. (1995). Fractional Bayes factors for model comparison (with discussion). Journal of the Royal Statistical Society, Series B, 57, 99–138MathSciNetMATHGoogle Scholar
  17. Raftery, A., Madigan, D., Hoeting, J. (1997). Bayesian model averaging for linear regression models. Journal of the American Statistical Association, 92 179–191.MathSciNetMATHCrossRefGoogle Scholar
  18. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.MathSciNetMATHCrossRefGoogle Scholar
  19. Smith, M. and Kohn, R. (1996). Nonparametric regression using Bayesian variable selection. J. Econometrics, 75, 317–344.MATHCrossRefGoogle Scholar
  20. Smith, M., and Kohn, R. (1997). A Bayesian approach to nonparametric bivariate regression. Journal of the American Statistical Association, 92,1522–1535.MathSciNetMATHCrossRefGoogle Scholar
  21. Smith, M., Wong, C. and Kohn, R. (1998). Additive nonparametric regression with autocorrelated errors. Journal of the Royal Statistical Society, Ser. B, in press.Google Scholar
  22. Stone, C. (1994) The use of polynomial splines and their tensor products in multivariate function estimation. (with discussion) The Annals of Statistics, 22 118–184.MathSciNetMATHCrossRefGoogle Scholar
  23. Tierney, L. (1994). Markov chains for exploring posterior distributions. The Annals of Statistics, 22 1701–1762.MathSciNetMATHCrossRefGoogle Scholar
  24. Wahba, G. (1990). Spline models for observational data, SIAM: Philadelphia.MATHCrossRefGoogle Scholar
  25. Wong, F., Hansen, M., Kohn, R., Smith, M. (1997).‘Focused sampling and its application to nonparametric regression’, preprint.Google Scholar

Copyright information

© Springer Science+Business Media New York 1998

Authors and Affiliations

  • Michael Smith
  • Robert Kohn

There are no affiliations available

Personalised recommendations