A Review of Heteroscedasticity Treatment with Gaussian Processes and Quantile Regression Meta-models
For regression problems, the general practice is to consider a constant variance of the error term across all data. This aims to simplify an often complicated model and relies on the assumption that this error is independent of the input variables. This property is known as homoscedasticity. On the other hand, in the real world, this is often a naive assumption, as we are rarely able to exhaustively include all true explanatory variables for a regression. While Big Data is bringing new opportunities for regression applications, ignoring this limitation may lead to biased estimators and inaccurate confidence and prediction intervals.
This paper aims to study the treatment of non-constant variance in regression models, also known as heteroscedasticity. We apply two methodologies: integration of conditional variance within the regression model itself; treat the regression model as a black box and use a meta-model that analyzes the error separately. We compare the performance of both approaches using two heteroscedastic data sets.
Although accounting for heteroscedasticity in data increases the complexity of the models used, we show that it can greatly improve the quality of the predictions, and more importantly, it can provide a proper notion of uncertainty or “confidence” associated with those predictions. We also discuss the feasibility of the solutions in a Big Data context.
KeywordsHeteroscedasticity Gaussian processes Quantile regression Confidence intervals Prediction intervals
- Bishop CM (2006) Pattern recognition and machine learning. Springer, New YorkGoogle Scholar
- Boukouvalas A, Barillec R, Cornford D (2012) Gaussian process quantile regression using expectation propagation. In: Proceedings of the 29th international conference on machine learning (ICML-12), pp 1695–1702Google Scholar
- Chen C, Hu J, Meng T, Zhang Y (2011) Short-time traffic flow prediction with ARIMA-GARCH model. In: Intelligent vehicles symposium (IV), IEEE, pp 607–612Google Scholar
- Fargas JA, Ben-Akiva ME, Pereira FC (2014) Prediction interval modeling using gaussian process quantile regression. Master’s Thesis, MIT, pp 1–65Google Scholar
- Goldberg P, Williams C, Bishop C (1998) Regression with input-dependent noise: a Gaussian process treatment. Adv Neural Inf Process Syst 10:493–499Google Scholar
- Gredilla LG, Titsias MK (2012) Variational heteroscedastic Gaussian process regression. In: 28th international conference on machine learningGoogle Scholar
- Hensman J, Fusi N, Lawrence ND (2013) Gaussian processes for big data. In: Proceedings of the 29th conference annual conference on uncertainty in artificial intelligence (UAI-13), pp 282–290Google Scholar
- Kersting K, Plagemann C, Pfaff P, Burgard W (2007) Most likely heteroscedas-tic Gaussian process regression. In: Proceedings of the International Machine Learning Society, pp 393–400Google Scholar
- Long JS, Ervin LH (1998) Correcting for heteroscedasticity with heteroscedasticity-consistent standard errors in the linear regression model: small sample considerations, Working Paper, Department of Statistics, Indiana UniversityGoogle Scholar
- MacKinnon JG (2012) Thirty years of heteroskedasticity-robust inference, Working Papers, Queen’s University, Department of EconomicsGoogle Scholar
- MacKinnon JG, White H (1983) Some heteroskedasticity consistent covariance matrix estimators with improved finite sample properties, Working Papers, Queen’s University, Department of EconomicsGoogle Scholar
- Osborne J, Waters E (2002) Four assumptions of multiple regression that researchers should always test. Pract Assess Res Eval 8(2):1–9Google Scholar
- Quinonero-Candela J, Rasmussen CE, Williams CKI (2007) Approximation methods for Gaussian process regression, Large-scale kernel machines, pp 203–223Google Scholar
- Rasmussen CE, Williams C (2006) Gaussian processes for machine learning. MIT Press, Cambridge, MAGoogle Scholar
- Silverman BW (1985) Some aspect of the spline smoothing approach to non-parametric regression curve fitting. J R Stat Soc 47(1):1–52Google Scholar
- Snelson E, Ghahramani Z (2007) Local and global sparse Gaussian process approximations. In: International conference on artificial intelligence and statistics, pp 524–531Google Scholar
- Zhou B, He D, Sun Z (2006) Traffic predictability based on ARIMA/GARCH model. In: 2nd conference on next generation internet design and engineering, pp 207–214Google Scholar