Advertisement

A Review of Heteroscedasticity Treatment with Gaussian Processes and Quantile Regression Meta-models

  • Francisco AntunesEmail author
  • Aidan O’Sullivan
  • Filipe Rodrigues
  • Francisco Pereira
Chapter
Part of the Springer Geography book series (SPRINGERGEOGR)

Abstract

For regression problems, the general practice is to consider a constant variance of the error term across all data. This aims to simplify an often complicated model and relies on the assumption that this error is independent of the input variables. This property is known as homoscedasticity. On the other hand, in the real world, this is often a naive assumption, as we are rarely able to exhaustively include all true explanatory variables for a regression. While Big Data is bringing new opportunities for regression applications, ignoring this limitation may lead to biased estimators and inaccurate confidence and prediction intervals.

This paper aims to study the treatment of non-constant variance in regression models, also known as heteroscedasticity. We apply two methodologies: integration of conditional variance within the regression model itself; treat the regression model as a black box and use a meta-model that analyzes the error separately. We compare the performance of both approaches using two heteroscedastic data sets.

Although accounting for heteroscedasticity in data increases the complexity of the models used, we show that it can greatly improve the quality of the predictions, and more importantly, it can provide a proper notion of uncertainty or “confidence” associated with those predictions. We also discuss the feasibility of the solutions in a Big Data context.

Keywords

Heteroscedasticity Gaussian processes Quantile regression Confidence intervals Prediction intervals 

References

  1. Bishop CM (2006) Pattern recognition and machine learning. Springer, New YorkGoogle Scholar
  2. Boukouvalas A, Barillec R, Cornford D (2012) Gaussian process quantile regression using expectation propagation. In: Proceedings of the 29th international conference on machine learning (ICML-12), pp 1695–1702Google Scholar
  3. Breusch TS, Pagan AR (1979) A simple test for heteroscedasticity and random coefficient variation. Econometrica 47(5):1287–1294CrossRefGoogle Scholar
  4. Chen C, Hu J, Meng T, Zhang Y (2011) Short-time traffic flow prediction with ARIMA-GARCH model. In: Intelligent vehicles symposium (IV), IEEE, pp 607–612Google Scholar
  5. Chipman JS (2011) International encyclopedia of statistical science. Springer, Berlin, pp 577–582CrossRefGoogle Scholar
  6. Cook RD, Weisberg S (1983) Diagnostics for heteroscedasticity in regression. Biometrika 70(1):1–10CrossRefGoogle Scholar
  7. Engle RF (1982) Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica 50(4):987–1007CrossRefGoogle Scholar
  8. Fargas JA, Ben-Akiva ME, Pereira FC (2014) Prediction interval modeling using gaussian process quantile regression. Master’s Thesis, MIT, pp 1–65Google Scholar
  9. Fox CW, Roberts SJ (2012) A tutorial on variational Bayesian inference. Artif Intell Rev 38(2):85–95CrossRefGoogle Scholar
  10. Goldberg P, Williams C, Bishop C (1998) Regression with input-dependent noise: a Gaussian process treatment. Adv Neural Inf Process Syst 10:493–499Google Scholar
  11. Goldfeld SM, Quandt RE (1965) Some tests for homoscedasticity. J Am Stat Assoc 60:539–547CrossRefGoogle Scholar
  12. Gredilla LG, Titsias MK (2012) Variational heteroscedastic Gaussian process regression. In: 28th international conference on machine learningGoogle Scholar
  13. Hensman J, Fusi N, Lawrence ND (2013) Gaussian processes for big data. In: Proceedings of the 29th conference annual conference on uncertainty in artificial intelligence (UAI-13), pp 282–290Google Scholar
  14. Kersting K, Plagemann C, Pfaff P, Burgard W (2007) Most likely heteroscedas-tic Gaussian process regression. In: Proceedings of the International Machine Learning Society, pp 393–400Google Scholar
  15. Khosravi A, Mazloumi E, Nahavandi S, Creighton D, Van Lint JWC (2011) Prediction intervals to account for uncertainties in travel time prediction. IEEE Trans Intell Transp Syst 12(2):537–547CrossRefGoogle Scholar
  16. Koenker R, Hallock KF (2001) Quantile regression. J Econ Perspect 15(4):143–156CrossRefGoogle Scholar
  17. Lee YS, Scholtes S (2014) Empirical prediction intervals revisited. Int J Forecast 30(2):217–234CrossRefGoogle Scholar
  18. Leslie DS, Kohn R, Nott DJ (2007) A general approach to heteroscedastic linear regression. Stat Comput 17(2):131–146CrossRefGoogle Scholar
  19. Long JS, Ervin LH (1998) Correcting for heteroscedasticity with heteroscedasticity-consistent standard errors in the linear regression model: small sample considerations, Working Paper, Department of Statistics, Indiana UniversityGoogle Scholar
  20. MacKinnon JG (2012) Thirty years of heteroskedasticity-robust inference, Working Papers, Queen’s University, Department of EconomicsGoogle Scholar
  21. MacKinnon JG, White H (1983) Some heteroskedasticity consistent covariance matrix estimators with improved finite sample properties, Working Papers, Queen’s University, Department of EconomicsGoogle Scholar
  22. Osborne J, Waters E (2002) Four assumptions of multiple regression that researchers should always test. Pract Assess Res Eval 8(2):1–9Google Scholar
  23. Pereira FC, Antoniou C, Fargas C, Ben-Akiva M (2014) A meta-model for estimating error bounds in real-traffic prediction systems. IEEE Trans Intell Trans Syst 15:1–13CrossRefGoogle Scholar
  24. Quinonero-Candela J, Rasmussen CE, Williams CKI (2007) Approximation methods for Gaussian process regression, Large-scale kernel machines, pp 203–223Google Scholar
  25. Rasmussen CE, Williams C (2006) Gaussian processes for machine learning. MIT Press, Cambridge, MAGoogle Scholar
  26. Robinson PM (1987) Asymptotically efficient estimation in the presence of heteroskedasticity of unknown form. Econometrica 55(4):875–891CrossRefGoogle Scholar
  27. Silverman BW (1985) Some aspect of the spline smoothing approach to non-parametric regression curve fitting. J R Stat Soc 47(1):1–52Google Scholar
  28. Snelson E, Ghahramani Z (2007) Local and global sparse Gaussian process approximations. In: International conference on artificial intelligence and statistics, pp 524–531Google Scholar
  29. Taylor JW, Bunn DW (1999) A quantile regression approach to generating prediction intervals. Manag Sci 45(2):225–237CrossRefGoogle Scholar
  30. Tsekeris T, Stathopoulos A (2006) Real-time traffic volatility forecasting in urban arterial networks. Transp Res Rec 1964:146–156CrossRefGoogle Scholar
  31. Tzikas DG, Likas AC, Galatsanos NP (2008) The variational approximation for Bayesian inference. IEEE Signal Process Mag 25(6):131–146CrossRefGoogle Scholar
  32. White H (1980) A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48(4):817–838CrossRefGoogle Scholar
  33. Zeileis A, Wien W (2004) Econometric computing with HC and HAC covariance matrix estimators. J Stat Softw 11(10):1–17CrossRefGoogle Scholar
  34. Zhou B, He D, Sun Z (2006) Traffic predictability based on ARIMA/GARCH model. In: 2nd conference on next generation internet design and engineering, pp 207–214Google Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Francisco Antunes
    • 1
    Email author
  • Aidan O’Sullivan
    • 2
  • Filipe Rodrigues
    • 3
  • Francisco Pereira
    • 3
  1. 1.Center of Informatics and Systems of the University of CoimbraCoimbraPortugal
  2. 2.Singapore-MIT Alliance for Research and TechnologySingaporeSingapore
  3. 3.Technical University of DenmarkLyngbyDenmark

Personalised recommendations